Sep 17 21:56:37.429544 ip-10-0-142-197 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 21:56:37.429556 ip-10-0-142-197 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 21:56:37.429563 ip-10-0-142-197 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 21:56:37.429700 ip-10-0-142-197 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 21:56:47.652680 ip-10-0-142-197 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 21:56:47.652698 ip-10-0-142-197 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 44105e4c6602402488b4cea12f30bfce --
Sep 17 21:58:40.984301 ip-10-0-142-197 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 21:58:41.402814 ip-10-0-142-197 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:41.402814 ip-10-0-142-197 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 21:58:41.402814 ip-10-0-142-197 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:41.402814 ip-10-0-142-197 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 21:58:41.402814 ip-10-0-142-197 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:41.404533 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.404453    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 21:58:41.407596 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407577    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:41.407596 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407592    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:41.407596 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407597    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:41.407596 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407602    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407607    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407611    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407616    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407620    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407623    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407637    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407644    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407649    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407653    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407657    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407661    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407665    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407670    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407674    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407677    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407681    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407685    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407689    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:41.407819 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407693    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407697    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407701    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407705    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407709    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407713    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407717    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407721    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407725    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407729    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407734    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407738    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407742    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407746    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407750    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407755    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407759    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407763    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407768    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:41.408457 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407772    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407776    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407780    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407784    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407789    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407793    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407797    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407801    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407805    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407809    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407813    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407819    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407824    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407829    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407833    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407837    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407841    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407845    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407849    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407853    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:41.409005 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407858    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407862    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407866    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407869    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407874    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407879    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407884    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407888    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407892    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407896    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407902    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407906    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407910    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407914    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407919    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407923    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407927    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407931    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407935    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407939    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:41.409811 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407943    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407948    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407951    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407955    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407959    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407963    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407967    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407974    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407978    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.407983    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.408003    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.408008    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.408012    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.408016    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.408020    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.408026    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.408032    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.408037    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409855    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409865    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:41.410740 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409870    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409874    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409878    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409884    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409889    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409894    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409898    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409903    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409907    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409911    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409915    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409918    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409923    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409927    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409931    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409935    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409939    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409944    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409948    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409952    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:41.411580 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409955    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409960    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409964    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409968    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409972    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409976    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409980    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.409984    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410007    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410011    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410015    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410020    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410023    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410027    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410032    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410035    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410041    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410045    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410049    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:41.412514 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410053    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410057    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410061    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410067    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410073    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410077    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410082    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410087    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410091    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410096    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410100    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410106    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410111    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410117    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410122    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410127    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410131    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410135    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410139    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:41.413194 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410143    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410147    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410151    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410155    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410167    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410172    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410176    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410180    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410184    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410188    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410192    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410198    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410202    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410206    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410210    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410214    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410218    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410223    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410227    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410231    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:41.413742 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410236    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410240    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410245    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410249    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410253    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410258    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410262    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410266    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410270    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410274    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410278    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410282    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410286    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410290    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410294    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410298    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410303    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410308    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410314    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410847    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:41.414281 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410856    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410861    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410866    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410871    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410876    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410880    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410884    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410888    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410892    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410897    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410901    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410905    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410912    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410918    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410924    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410928    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410933    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410937    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410942    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:41.414767 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410947    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410952    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410956    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410961    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410965    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410970    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410974    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410978    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410982    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.410986    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411007    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411011    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411015    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411019    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411024    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411028    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411032    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411037    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411041    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411045    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:41.415257 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411049    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411053    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411057    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411061    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411066    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411070    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411074    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411081    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411086    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411090    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411096    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411100    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411105    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411109    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411113    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411117    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411121    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411126    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411130    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:41.415799 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411134    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411138    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411142    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411146    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411150    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411154    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411158    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411163    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411167    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411171    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411176    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411180    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411184    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411189    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411192    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411196    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411200    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411206    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411211    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:41.416399 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411215    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411219    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411224    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411228    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411232    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411235    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411239    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411244    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411248    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411252    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411256    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411262    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411268    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411273    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411277    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411282    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411286    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411290    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411294    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411299    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:41.416851 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.411303    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411400    2567 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411446    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411457    2567 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411463    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411470    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411475    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411482    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411488    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411492    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411497    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411503    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411507    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411512    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411517    2567 flags.go:64] FLAG: --cgroup-root=""
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411522    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411526    2567 flags.go:64] FLAG: --client-ca-file=""
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411531    2567 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411535    2567 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411541    2567 flags.go:64] FLAG: --cluster-domain=""
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411546    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411551    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411556    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411562    2567 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 21:58:41.417436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411567    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411572    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411577    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411582    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411587    2567 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411591    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411595    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411600    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411604    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411610    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411615    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411620    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411624    2567 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411629    2567 flags.go:64] FLAG: --enable-server="true"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411635    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411642    2567 flags.go:64] FLAG: --event-burst="100"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411647    2567 flags.go:64] FLAG: --event-qps="50"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411651    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411657    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411662    2567 flags.go:64] FLAG: --eviction-hard=""
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411674    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411679    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411684    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411688    2567 flags.go:64] FLAG: --eviction-soft=""
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411693    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 21:58:41.418027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411697    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411702    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411707    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411711    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411716    2567 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411721    2567 flags.go:64] FLAG: --feature-gates=""
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411726    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411731    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411736    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411741    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411745    2567 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411750    2567 flags.go:64] FLAG: --help="false"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411755    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411759    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411764    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411768    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411774    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411779    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411784    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411789    2567 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411793    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411798    2567 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411803    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411808    2567 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 21:58:41.419094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411813    2567 flags.go:64] FLAG: --kube-reserved=""
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411817    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411824    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411829    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411834    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411838    2567 flags.go:64] FLAG: --lock-file=""
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411842    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411847    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411852    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411860    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411865    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411870    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411874    2567 flags.go:64] FLAG: --logging-format="text"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411879    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411884    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411889    2567 flags.go:64] FLAG: --manifest-url=""
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411893    2567 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411899    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411904    2567 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411910    2567 flags.go:64] FLAG: --max-pods="110"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411915    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411920    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411924    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411929    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411933    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 21:58:41.420113 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411938    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411943    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411953    2567 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411958    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411962    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411966    2567 flags.go:64] FLAG: --pod-cidr=""
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411972    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411980    2567 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.411985    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412006    2567 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412015    2567 flags.go:64] FLAG: --port="10250"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412020    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412024    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-001cb0df60f91fa26"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412028    2567 flags.go:64] FLAG: --qos-reserved=""
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412033    2567 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412038    2567 flags.go:64] FLAG: --register-node="true"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412042    2567 flags.go:64] FLAG: --register-with-taints=""
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412049    2567 flags.go:64] FLAG: --registry-burst="10"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412054    2567 flags.go:64] FLAG: --registry-qps="5"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412058    2567 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412063    2567 flags.go:64] FLAG: --reserved-memory=""
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412069    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412074    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412079    2567 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412084    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 21:58:41.420937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412089    2567 flags.go:64] FLAG: --runonce="false"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412094    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412098    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412104    2567 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412108    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412112    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412117    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412122    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412127    2567 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412131    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412136    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412140    2567 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412144    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412149    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412155    2567 flags.go:64] FLAG: --system-cgroups=""
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412169    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412178    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412185    2567 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412190    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412196    2567 flags.go:64] FLAG: --tls-min-version=""
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412200    2567 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412204    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412209    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412215    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412244    2567 flags.go:64] FLAG: --v="2"
Sep 17 21:58:41.421582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412251    2567 flags.go:64] FLAG: --version="false"
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412257    2567 flags.go:64] FLAG: --vmodule=""
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412263    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.412268    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412441    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412448    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412453    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412458    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412462    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412467    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412471    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412475    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412479    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412483    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412487    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412492    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412496    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412500    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412504    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412509    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:41.422216 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412513    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412517    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412521    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412525    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412532    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412536    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412540    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412544    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412548    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412553    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412559    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412564    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412568    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412572    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412576    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412580    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412584    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412588    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412593    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412597    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:41.422702 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412602    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412605    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412609    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412613    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412618    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412622    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412626    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412630    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412634    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412638    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412643    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412647    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412651    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412655    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412659    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412666    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412675    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412680    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412685    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412690    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:41.423240 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412694    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412699    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412705    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412709    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412713    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412717    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412721    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412726    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412730    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412734    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412738    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412745    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412750    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412756    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412761    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412766    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412770    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412774    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412778    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:41.423927 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412783    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412787    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412791    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412795    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412799    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412804    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412808    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412812    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412816    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412823    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412827    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412831    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412835    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412839    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412843    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412850    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412854    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412858    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412863    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:41.424662 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412867    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:41.425163 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412873    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:41.425163 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412878    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:41.425163 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412882    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:41.425163 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.412886    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:41.425163 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.413478    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:41.425163 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.422538    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 21:58:41.425163 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.422556    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 21:58:41.425163 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.422575    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 21:58:41.425163 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.422583    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 21:58:41.425163 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422650    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:41.425163 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422655    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:41.425163 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422658    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:41.425163 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422661    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:41.425163 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422664    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422666    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422669    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422672    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422674    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422677    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422680    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422682    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422685    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422688    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422691    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422693    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422696    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422699    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422702    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422704    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422707    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422710    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422713    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422715    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:41.425503 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422718    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422721    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422723    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422726    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422730    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422740    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422744    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422746    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422749    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422752    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422754    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422758    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422762    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422765    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422768    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422771    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422774    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422777    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422780    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:41.425983 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422782    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422785    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422787    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422790    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422798    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422801    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422804    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422806    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422809    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422811    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422814    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422817    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422819    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422822    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422825    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422827    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422830    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422833    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422836    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422847    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:41.426473 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422850    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422853    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422855    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422858    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422861    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422863    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422866    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422869    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422871    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422874    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422876    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422879    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422881    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422884    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422886    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422889    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422892    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422894    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422897    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422900    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:41.426967 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422902    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422905    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422907    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422910    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422912    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422915    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422917    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422920    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422923    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422926    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422929    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422931    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422940    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422944    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422947    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:41.427501 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.422950    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.422956    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423124    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423129    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423132    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423135    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423138    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423141    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423144    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423146    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423149    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423151    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423154    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423157    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423159    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:41.427858 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423162    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423164    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423167    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423169    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423172    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423175    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423177    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423179    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423182    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423185    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423188    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423190    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423193    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423196    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423198    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423207    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423209    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423212    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423214    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423217    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:41.428248 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423220    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423222    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423224    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423227    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423230    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423232    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423235    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423237    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423240    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423243    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423245    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423248    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423251    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423254    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423256    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423259    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423262    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423264    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423267    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423269    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:41.428755 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423272    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423275    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423277    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423280    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423282    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423285    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423287    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423290    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423299    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423302    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423304    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423307    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423310    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423312    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423315    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423317    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423321    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423325    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:41.429358 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423327    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423330    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423333    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423336    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423339    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423342    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423345    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423347    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423351    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423354    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423358    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423360    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423363    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423366    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423369    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423373    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423376    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423379    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423381    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:41.429800 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423384    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:41.430295 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423386    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:41.430295 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423389    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:41.430295 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423391    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:41.430295 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423400    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:41.430295 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423403    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:41.430295 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423405    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:41.430295 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423408    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:41.430295 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:41.423410    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:41.430295 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.423415    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:41.430295 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.424959    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 21:58:41.430295 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.427098    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 21:58:41.430295 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.428191    2567 server.go:1019] "Starting client certificate rotation"
Sep 17 21:58:41.430295 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.428318    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:58:41.430295 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.428354    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:58:41.430875 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.430861    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:58:41.432013 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.431983    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:58:41.440800 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.440786    2567 log.go:25] "Validated CRI v1 runtime API"
Sep 17 21:58:41.442323 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.442306    2567 log.go:25] "Validated CRI v1 image API"
Sep 17 21:58:41.443183 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.443167    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 21:58:41.448511 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.448492    2567 fs.go:135] Filesystem UUIDs: map[1e9886bc-f8db-4952-bd77-3708a6195114:/dev/nvme0n1p4 5f9d4244-85a4-4a68-9d40-3f5ea13c66cc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 17 21:58:41.448587 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.448512    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 21:58:41.453818 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.453707    2567 manager.go:216] Machine: {Timestamp:2026-09-17 21:58:41.4524088 +0000 UTC m=+0.348346574 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3187442 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e2ec85b23693d59d514672c35e391 SystemUUID:ec2e2ec8-5b23-693d-59d5-14672c35e391 BootID:44105e4c-6602-4024-88b4-cea12f30bfce Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cb:34:42:61 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cb:34:42:61 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:db:e9:03:47:f7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 21:58:41.453818 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.453811    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 21:58:41.453938 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.453910    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 21:58:41.456982 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.456917    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 21:58:41.457302 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.456986    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-197.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 21:58:41.458021 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.458005    2567 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 21:58:41.458077 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.458028    2567 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 21:58:41.458077 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.458040    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:58:41.458819 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.458781    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:58:41.459039 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.459026    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 21:58:41.461927 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.461911    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:58:41.462696 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.462684    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:58:41.462813 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.462803    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 21:58:41.462844 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.462834    2567 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 21:58:41.462871 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.462848    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 21:58:41.462871 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.462869    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 21:58:41.462926 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.462883    2567 kubelet.go:398] "Adding apiserver pod source"
Sep 17 21:58:41.462926 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.462896    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 21:58:41.463337 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.463323    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:58:41.463413 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.463346    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:58:41.465471 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.465458    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 21:58:41.467745 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.467731    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 21:58:41.467803 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.467760    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 21:58:41.470419 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.470401    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 21:58:41.470490 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.470423    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 21:58:41.470490 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.470430    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 21:58:41.470490 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.470435    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 21:58:41.470490 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.470444    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 21:58:41.470490 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.470449    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 21:58:41.470490 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.470462    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 21:58:41.470490 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.470467    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 21:58:41.470490 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.470473    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 21:58:41.470490 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.470479    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 21:58:41.470727 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.470500    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 21:58:41.470727 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.470506    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 21:58:41.471248 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.471235    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 21:58:41.471248 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.471245    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 21:58:41.473876 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.473847    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 21:58:41.474689 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.474676    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-197.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 21:58:41.475119 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.475106    2567 server.go:1268] "Started kubelet"
Sep 17 21:58:41.475266 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.475221    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 21:58:41.475300 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.475263    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 21:58:41.475300 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.475288    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 21:58:41.475387 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.475164    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 21:58:41.475745 ip-10-0-142-197 systemd[1]: Started Kubernetes Kubelet.
Sep 17 21:58:41.477129 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.477109    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 21:58:41.477204 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.477158    2567 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 21:58:41.481814 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.481793    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 21:58:41.484724 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.484704    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 21:58:41.484815 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.484724    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 21:58:41.484815 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.484774    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 17 21:58:41.485355 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.485330    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 21:58:41.485454 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.485332    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 21:58:41.485454 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.485374    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 21:58:41.485454 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.485442    2567 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 21:58:41.485454 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.485451    2567 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 21:58:41.485615 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.485578    2567 factory.go:55] Registering systemd factory
Sep 17 21:58:41.485615 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.485610    2567 factory.go:223] Registration of the systemd container factory successfully
Sep 17 21:58:41.485880 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.485807    2567 factory.go:153] Registering CRI-O factory
Sep 17 21:58:41.485880 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.485820    2567 factory.go:223] Registration of the crio container factory successfully
Sep 17 21:58:41.485880 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.485866    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 21:58:41.485880 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.485883    2567 factory.go:103] Registering Raw factory
Sep 17 21:58:41.486112 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.485896    2567 manager.go:1196] Started watching for new ooms in manager
Sep 17 21:58:41.486281 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.486268    2567 manager.go:349] Starting recovery of all containers
Sep 17 21:58:41.491931 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.491904    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 21:58:41.492124 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.492102    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-197.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 21:58:41.493505 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.491920    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-197.ec2.internal.18d63a96d8513f2f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-197.ec2.internal,UID:ip-10-0-142-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-197.ec2.internal,},FirstTimestamp:2026-09-17 21:58:41.475059503 +0000 UTC m=+0.370997276,LastTimestamp:2026-09-17 21:58:41.475059503 +0000 UTC m=+0.370997276,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-197.ec2.internal,}"
Sep 17 21:58:41.498749 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.498733    2567 manager.go:354] Recovery completed
Sep 17 21:58:41.505336 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.505325    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:41.506903 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.506873    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:41.506978 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.506918    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:41.506978 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.506931    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:41.507469 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.507456    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 21:58:41.507513 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.507470    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 21:58:41.507513 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.507487    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:58:41.507567 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.507460    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:41.508193 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.508174    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:41.508279 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.508206    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:41.508279 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.508219    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:41.508807 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.508749    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-197.ec2.internal.18d63a96da371f8f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-197.ec2.internal,UID:ip-10-0-142-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-197.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-197.ec2.internal,},FirstTimestamp:2026-09-17 21:58:41.506901903 +0000 UTC m=+0.402839684,LastTimestamp:2026-09-17 21:58:41.506901903 +0000 UTC m=+0.402839684,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-197.ec2.internal,}"
Sep 17 21:58:41.509039 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.509025    2567 policy_none.go:49] "None policy: Start"
Sep 17 21:58:41.509085 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.509042    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 21:58:41.509085 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.509051    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 21:58:41.510128 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.510117    2567 policy_none.go:47] "Start"
Sep 17 21:58:41.519230 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.519212    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9kbkm"
Sep 17 21:58:41.519361 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.519272    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-197.ec2.internal.18d63a96da377a95  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-197.ec2.internal,UID:ip-10-0-142-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-197.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-197.ec2.internal,},FirstTimestamp:2026-09-17 21:58:41.506925205 +0000 UTC m=+0.402862979,LastTimestamp:2026-09-17 21:58:41.506925205 +0000 UTC m=+0.402862979,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-197.ec2.internal,}"
Sep 17 21:58:41.526036 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.525981    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9kbkm"
Sep 17 21:58:41.529979 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.529917    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-197.ec2.internal.18d63a96da37a108  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-197.ec2.internal,UID:ip-10-0-142-197.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-197.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-197.ec2.internal,},FirstTimestamp:2026-09-17 21:58:41.506935048 +0000 UTC m=+0.402872822,LastTimestamp:2026-09-17 21:58:41.506935048 +0000 UTC m=+0.402872822,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-197.ec2.internal,}"
Sep 17 21:58:41.531644 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.531629    2567 manager.go:337] "Starting Device Plugin manager"
Sep 17 21:58:41.531716 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.531674    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 21:58:41.531716 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.531686    2567 server.go:87] "Starting device plugin registration server"
Sep 17 21:58:41.531905 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.531893    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 21:58:41.531963 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.531906    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 21:58:41.532090 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.532070    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 21:58:41.532436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.532151    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 21:58:41.532436 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.532160    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 21:58:41.532544 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.532534    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 21:58:41.532594 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.532582    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 17 21:58:41.551202 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.551186    2567 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2601 into the system container \"/system.slice\": <nil>"
Sep 17 21:58:41.632832 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.632792    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:41.634917 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.634901    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:41.635016 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.634930    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:41.635016 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.634945    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:41.635016 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.634972    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.645975 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.645956    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.645975 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.645975    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-197.ec2.internal\": node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 17 21:58:41.663217 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.663183    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 17 21:58:41.746177 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.746148    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 21:58:41.747261 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.747241    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 21:58:41.747340 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.747285    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 21:58:41.747340 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.747303    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 21:58:41.747782 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.747337    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 21:58:41.749979 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.749959    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:41.763271 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.763254    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 17 21:58:41.849346 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.848644    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"]
Sep 17 21:58:41.849419 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.849392    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:41.850268 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.850248    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:41.850353 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.850283    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:41.850353 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.850295    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:41.851696 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.851684    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:41.851806 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.851791    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.851868 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.851818    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:41.852384 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.852369    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:41.852384 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.852377    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:41.852487 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.852399    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:41.852487 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.852410    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:41.852487 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.852416    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:41.852487 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.852442    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:41.853832 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.853819    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.853884 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.853841    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:41.854488 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.854471    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:41.854538 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.854499    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:41.854538 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.854508    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:41.863704 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.863691    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 17 21:58:41.877634 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.877615    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-197.ec2.internal\" not found" node="ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.881957 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.881943    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-197.ec2.internal\" not found" node="ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.887170 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.887155    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d19258485fc6925eb879b1d7211e2fc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal\" (UID: \"d19258485fc6925eb879b1d7211e2fc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.887230 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.887177    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d19258485fc6925eb879b1d7211e2fc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal\" (UID: \"d19258485fc6925eb879b1d7211e2fc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.887230 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.887194    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6a12a94a946441191f85a154c16cabf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-197.ec2.internal\" (UID: \"a6a12a94a946441191f85a154c16cabf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.964631 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:41.964584    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 17 21:58:41.987948 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.987930    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6a12a94a946441191f85a154c16cabf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-197.ec2.internal\" (UID: \"a6a12a94a946441191f85a154c16cabf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.988023 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.987954    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d19258485fc6925eb879b1d7211e2fc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal\" (UID: \"d19258485fc6925eb879b1d7211e2fc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.988023 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.987970    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d19258485fc6925eb879b1d7211e2fc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal\" (UID: \"d19258485fc6925eb879b1d7211e2fc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.988117 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.988028    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d19258485fc6925eb879b1d7211e2fc7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal\" (UID: \"d19258485fc6925eb879b1d7211e2fc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.988117 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.988033    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6a12a94a946441191f85a154c16cabf-config\") pod \"kube-apiserver-proxy-ip-10-0-142-197.ec2.internal\" (UID: \"a6a12a94a946441191f85a154c16cabf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:41.988117 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:41.988064    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d19258485fc6925eb879b1d7211e2fc7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal\" (UID: \"d19258485fc6925eb879b1d7211e2fc7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:42.065059 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:42.065034    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 17 21:58:42.165585 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:42.165561    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 17 21:58:42.181710 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:42.181696    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:44.073316 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.073053    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:44.153263 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.153213    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 17 21:58:44.253756 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.253735    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-197.ec2.internal\" not found"
Sep 17 21:58:44.313037 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.313019    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:44.315779 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.315767    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 21:58:44.315875 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.315857    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 21:58:44.315926 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.315905    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 21:58:44.351065 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.351046    2567 apiserver.go:52] "Watching apiserver"
Sep 17 21:58:44.368557 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.368539    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 21:58:44.368851 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.368834    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-th2bd","openshift-cluster-node-tuning-operator/tuned-v76w8","openshift-dns/node-resolver-n4zhm","openshift-network-operator/iptables-alerter-8k6br","openshift-ovn-kubernetes/ovnkube-node-2kp5t","kube-system/global-pull-secret-syncer-58crn","kube-system/konnectivity-agent-qhmn8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng","openshift-image-registry/node-ca-xc5tg","openshift-multus/multus-additional-cni-plugins-q9nmn","openshift-multus/multus-q79d2","openshift-multus/network-metrics-daemon-fdxms"]
Sep 17 21:58:44.370238 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.370225    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:58:44.370296 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.370280    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:58:44.371262 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.371248    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.372023 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.371989    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:44.372097 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.372021    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 21:58:44.372323 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.372308    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n4zhm"
Sep 17 21:58:44.373415 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.373397    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 21:58:44.373523 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.373506    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8k6br"
Sep 17 21:58:44.373523 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.373518    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n687c\""
Sep 17 21:58:44.373664 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.373646    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:44.374759 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.374745    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.375133 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.375113    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 21:58:44.375349 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.375337    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 21:58:44.375409 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.375396    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fjdmf\""
Sep 17 21:58:44.375530 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.375519    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:44.375830 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.375818    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 21:58:44.375916 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.375905    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:44.375980 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.375964    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:58:44.376045 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.376014    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5r7hz\""
Sep 17 21:58:44.376109 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.376093    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 21:58:44.378180 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.377837    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwxxx\" (UniqueName: \"kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx\") pod \"network-check-target-th2bd\" (UID: \"da365699-cc35-49b9-a86a-58abbcc127f2\") " pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:58:44.378180 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.377881    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-lib-modules\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.378180 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.377912    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2218d4e3-f6ca-4473-8187-abc582397c27-etc-tuned\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.378180 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.377942    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/149b4a0e-4466-46a6-9e51-d2450702c4d0-tmp-dir\") pod \"node-resolver-n4zhm\" (UID: \"149b4a0e-4466-46a6-9e51-d2450702c4d0\") " pod="openshift-dns/node-resolver-n4zhm"
Sep 17 21:58:44.378180 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.377972    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f58g\" (UniqueName: \"kubernetes.io/projected/0b2f2051-ca1f-4ba1-897c-7266d76556c4-kube-api-access-8f58g\") pod \"iptables-alerter-8k6br\" (UID: \"0b2f2051-ca1f-4ba1-897c-7266d76556c4\") " pod="openshift-network-operator/iptables-alerter-8k6br"
Sep 17 21:58:44.378180 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.377995    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-modprobe-d\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.378180 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378065    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-sysctl-d\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.378180 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-sysctl-conf\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.378180 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378122    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-run\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.378180 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378147    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-sys\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378268    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-var-lib-kubelet\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378349    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2218d4e3-f6ca-4473-8187-abc582397c27-tmp\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378404    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/149b4a0e-4466-46a6-9e51-d2450702c4d0-hosts-file\") pod \"node-resolver-n4zhm\" (UID: \"149b4a0e-4466-46a6-9e51-d2450702c4d0\") " pod="openshift-dns/node-resolver-n4zhm"
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378444    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-sysconfig\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378464    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378471    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-kubernetes\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378466    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378481    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378519    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-systemd\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378526    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d2bsb\""
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378547    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-host\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378577    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hsb9\" (UniqueName: \"kubernetes.io/projected/2218d4e3-f6ca-4473-8187-abc582397c27-kube-api-access-5hsb9\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378602    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378531    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378626    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kfk8\" (UniqueName: \"kubernetes.io/projected/149b4a0e-4466-46a6-9e51-d2450702c4d0-kube-api-access-8kfk8\") pod \"node-resolver-n4zhm\" (UID: \"149b4a0e-4466-46a6-9e51-d2450702c4d0\") " pod="openshift-dns/node-resolver-n4zhm"
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378653    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b2f2051-ca1f-4ba1-897c-7266d76556c4-iptables-alerter-script\") pod \"iptables-alerter-8k6br\" (UID: \"0b2f2051-ca1f-4ba1-897c-7266d76556c4\") " pod="openshift-network-operator/iptables-alerter-8k6br"
Sep 17 21:58:44.378704 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.378683    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b2f2051-ca1f-4ba1-897c-7266d76556c4-host-slash\") pod \"iptables-alerter-8k6br\" (UID: \"0b2f2051-ca1f-4ba1-897c-7266d76556c4\") " pod="openshift-network-operator/iptables-alerter-8k6br"
Sep 17 21:58:44.379296 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.379140    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhmn8"
Sep 17 21:58:44.379296 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.379282    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 21:58:44.380758 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.380743    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.381194 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.381179    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 21:58:44.381368 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.381357    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 21:58:44.381482 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.381468    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qfrds\""
Sep 17 21:58:44.382103 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.382091    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xc5tg"
Sep 17 21:58:44.382886 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.382871    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 21:58:44.383019 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.382990    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 21:58:44.383081 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.383021    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:58:44.383121 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.383099    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"
Sep 17 21:58:44.383151 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.383131    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n2kq8\""
Sep 17 21:58:44.383304 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.383287    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 21:58:44.383536 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.383522    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.383968 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.383952    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 21:58:44.384597 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.384584    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 21:58:44.384690 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.384675    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 21:58:44.384846 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.384828    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.385018 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.384977    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rbvld\""
Sep 17 21:58:44.385690 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.385678    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 21:58:44.385867 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.385851    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 21:58:44.385948 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.385851    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zxx5t\""
Sep 17 21:58:44.385948 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.385894    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 21:58:44.386181 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.386165    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 21:58:44.386230 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.386191    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal"]
Sep 17 21:58:44.386330 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.386166    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 21:58:44.386330 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.386267    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:44.386425 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.386323    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:58:44.386824 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.386811    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wrddd\""
Sep 17 21:58:44.387056 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.387042    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 21:58:44.388216 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.388090    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:58:44.392882 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.392865    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal"]
Sep 17 21:58:44.393033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.393017    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:58:44.411836 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.411821    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2p8s2"
Sep 17 21:58:44.417596 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.417556    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 21:53:43 +0000 UTC" deadline="2028-03-29 01:59:58.684519975 +0000 UTC"
Sep 17 21:58:44.417596 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.417594    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13396h1m14.266928015s"
Sep 17 21:58:44.419888 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.419875    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2p8s2"
Sep 17 21:58:44.473644 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.473618    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 21:58:44.479367 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479352    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-sysctl-d\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.479430 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479374    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-sysctl-conf\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.479430 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479391    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2218d4e3-f6ca-4473-8187-abc582397c27-tmp\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.479430 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479411    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kfk8\" (UniqueName: \"kubernetes.io/projected/149b4a0e-4466-46a6-9e51-d2450702c4d0-kube-api-access-8kfk8\") pod \"node-resolver-n4zhm\" (UID: \"149b4a0e-4466-46a6-9e51-d2450702c4d0\") " pod="openshift-dns/node-resolver-n4zhm"
Sep 17 21:58:44.479538 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479436    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-slash\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.479538 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479456    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-etc-openvswitch\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.479538 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479504    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.479538 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479513    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-sysctl-d\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.479538 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479537    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-sysctl-conf\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.479748 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479548    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-524vk\" (UniqueName: \"kubernetes.io/projected/94979e3c-1ac8-4800-a60f-3d3552644bcc-kube-api-access-524vk\") pod \"network-metrics-daemon-fdxms\" (UID: \"94979e3c-1ac8-4800-a60f-3d3552644bcc\") " pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:44.479748 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479568    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-env-overrides\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.479748 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479582    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-etc-selinux\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.479748 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479598    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd4fee5c-7168-471c-b29a-f92b77f59b67-multus-daemon-config\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.479748 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479638    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-etc-kubernetes\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.479748 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479692    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d856ebe3-2d7c-4833-9576-504511abb585-system-cni-dir\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.479748 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479725    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-run-netns\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.480023 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479759    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-ovn-node-metrics-cert\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.480023 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479782    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5dd82d87-0321-4c10-a183-6d6122eda0cc-serviceca\") pod \"node-ca-xc5tg\" (UID: \"5dd82d87-0321-4c10-a183-6d6122eda0cc\") " pod="openshift-image-registry/node-ca-xc5tg"
Sep 17 21:58:44.480023 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479820    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2218d4e3-f6ca-4473-8187-abc582397c27-etc-tuned\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.480023 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479857    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/149b4a0e-4466-46a6-9e51-d2450702c4d0-tmp-dir\") pod \"node-resolver-n4zhm\" (UID: \"149b4a0e-4466-46a6-9e51-d2450702c4d0\") " pod="openshift-dns/node-resolver-n4zhm"
Sep 17 21:58:44.480023 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479912    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:44.480023 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479914    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 21:58:44.480023 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479945    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-run\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.480023 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.479987    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-run\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480037    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-sys\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480058    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-var-lib-kubelet\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480074    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/149b4a0e-4466-46a6-9e51-d2450702c4d0-hosts-file\") pod \"node-resolver-n4zhm\" (UID: \"149b4a0e-4466-46a6-9e51-d2450702c4d0\") " pod="openshift-dns/node-resolver-n4zhm"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480105    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-run-ovn\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480129    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/149b4a0e-4466-46a6-9e51-d2450702c4d0-hosts-file\") pod \"node-resolver-n4zhm\" (UID: \"149b4a0e-4466-46a6-9e51-d2450702c4d0\") " pod="openshift-dns/node-resolver-n4zhm"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480133    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d856ebe3-2d7c-4833-9576-504511abb585-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480137    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-sys\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480177    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-var-lib-kubelet\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480180    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-systemd\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480208    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-host\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480232    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hsb9\" (UniqueName: \"kubernetes.io/projected/2218d4e3-f6ca-4473-8187-abc582397c27-kube-api-access-5hsb9\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480236    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-systemd\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480264    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-host\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480268    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-systemd-units\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480289    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-registration-dir\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.480327 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.480310    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b2f2051-ca1f-4ba1-897c-7266d76556c4-host-slash\") pod \"iptables-alerter-8k6br\" (UID: \"0b2f2051-ca1f-4ba1-897c-7266d76556c4\") " pod="openshift-network-operator/iptables-alerter-8k6br"
Sep 17 21:58:44.481135 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481114    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e8cae935-fc5f-46f9-ac7c-582f15caaf4c-agent-certs\") pod \"konnectivity-agent-qhmn8\" (UID: \"e8cae935-fc5f-46f9-ac7c-582f15caaf4c\") " pod="kube-system/konnectivity-agent-qhmn8"
Sep 17 21:58:44.481226 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481152    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-hostroot\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.481226 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481185    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs\") pod \"network-metrics-daemon-fdxms\" (UID: \"94979e3c-1ac8-4800-a60f-3d3552644bcc\") " pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:44.481226 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481200    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/149b4a0e-4466-46a6-9e51-d2450702c4d0-tmp-dir\") pod \"node-resolver-n4zhm\" (UID: \"149b4a0e-4466-46a6-9e51-d2450702c4d0\") " pod="openshift-dns/node-resolver-n4zhm"
Sep 17 21:58:44.481340 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481234    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-lib-modules\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.481340 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481267    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzp59\" (UniqueName: \"kubernetes.io/projected/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-kube-api-access-dzp59\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.481340 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481298    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfxrs\" (UniqueName: \"kubernetes.io/projected/2d5278a3-afa2-435f-ba85-6bd5658a483f-kube-api-access-gfxrs\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.481340 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481332    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-multus-conf-dir\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.481520 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481358    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcb7g\" (UniqueName: \"kubernetes.io/projected/bd4fee5c-7168-471c-b29a-f92b77f59b67-kube-api-access-wcb7g\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.481604 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481588    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-run-ovn-kubernetes\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.481660 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481631    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b2f2051-ca1f-4ba1-897c-7266d76556c4-host-slash\") pod \"iptables-alerter-8k6br\" (UID: \"0b2f2051-ca1f-4ba1-897c-7266d76556c4\") " pod="openshift-network-operator/iptables-alerter-8k6br"
Sep 17 21:58:44.481713 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481660    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-lib-modules\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.481713 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481668    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-ovnkube-script-lib\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.481713 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481703    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-sys-fs\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.481853 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481752    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45tmj\" (UniqueName: \"kubernetes.io/projected/d856ebe3-2d7c-4833-9576-504511abb585-kube-api-access-45tmj\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.481853 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481837    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-kubernetes\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.481954 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481871    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-run-k8s-cni-cncf-io\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.481954 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481929    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-run-netns\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.482068 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.481961    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-run-openvswitch\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.482117 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.482100    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e8cae935-fc5f-46f9-ac7c-582f15caaf4c-konnectivity-ca\") pod \"konnectivity-agent-qhmn8\" (UID: \"e8cae935-fc5f-46f9-ac7c-582f15caaf4c\") " pod="kube-system/konnectivity-agent-qhmn8"
Sep 17 21:58:44.482170 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.482139    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd4fee5c-7168-471c-b29a-f92b77f59b67-cni-binary-copy\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.482504 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.482338    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-kubelet-config\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:44.482504 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.482432    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-kubernetes\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.482504 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.482472    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d856ebe3-2d7c-4833-9576-504511abb585-cnibin\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.482504 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.482499    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d856ebe3-2d7c-4833-9576-504511abb585-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.482743 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.482529    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-kubelet\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.482743 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.482562    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxxx\" (UniqueName: \"kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx\") pod \"network-check-target-th2bd\" (UID: \"da365699-cc35-49b9-a86a-58abbcc127f2\") " pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:58:44.482743 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.482663    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-cni-bin\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.482968 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.482780    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-cni-netd\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.482968 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.482847    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-ovnkube-config\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.482968 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.482917    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-device-dir\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.482968 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.482946    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-multus-socket-dir-parent\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.483174 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.482980    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-var-lib-kubelet\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.483174 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483034    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b2f2051-ca1f-4ba1-897c-7266d76556c4-iptables-alerter-script\") pod \"iptables-alerter-8k6br\" (UID: \"0b2f2051-ca1f-4ba1-897c-7266d76556c4\") " pod="openshift-network-operator/iptables-alerter-8k6br"
Sep 17 21:58:44.483174 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-var-lib-openvswitch\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.483643 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483211    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-log-socket\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.483643 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483387    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.483643 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483433    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d856ebe3-2d7c-4833-9576-504511abb585-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.483643 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483491    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-sysconfig\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.483643 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483533    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-node-log\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.483643 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483556    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-system-cni-dir\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.483643 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483606    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b2f2051-ca1f-4ba1-897c-7266d76556c4-iptables-alerter-script\") pod \"iptables-alerter-8k6br\" (UID: \"0b2f2051-ca1f-4ba1-897c-7266d76556c4\") " pod="openshift-network-operator/iptables-alerter-8k6br"
Sep 17 21:58:44.483989 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483642    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-sysconfig\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.483989 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483626    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-multus-cni-dir\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.483989 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483703    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-var-lib-cni-bin\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.483989 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483738    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2218d4e3-f6ca-4473-8187-abc582397c27-tmp\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.483989 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483736    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-run-systemd\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.483989 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483783    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-cnibin\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.483989 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483789    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2218d4e3-f6ca-4473-8187-abc582397c27-etc-tuned\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.483989 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483827    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-var-lib-cni-multus\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.483989 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483870    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-run-multus-certs\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.483989 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483905    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd82d87-0321-4c10-a183-6d6122eda0cc-host\") pod \"node-ca-xc5tg\" (UID: \"5dd82d87-0321-4c10-a183-6d6122eda0cc\") " pod="openshift-image-registry/node-ca-xc5tg"
Sep 17 21:58:44.483989 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483937    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpgmz\" (UniqueName: \"kubernetes.io/projected/5dd82d87-0321-4c10-a183-6d6122eda0cc-kube-api-access-bpgmz\") pod \"node-ca-xc5tg\" (UID: \"5dd82d87-0321-4c10-a183-6d6122eda0cc\") " pod="openshift-image-registry/node-ca-xc5tg"
Sep 17 21:58:44.483989 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.483975    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f58g\" (UniqueName: \"kubernetes.io/projected/0b2f2051-ca1f-4ba1-897c-7266d76556c4-kube-api-access-8f58g\") pod \"iptables-alerter-8k6br\" (UID: \"0b2f2051-ca1f-4ba1-897c-7266d76556c4\") " pod="openshift-network-operator/iptables-alerter-8k6br"
Sep 17 21:58:44.484585 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.484023    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-socket-dir\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.484585 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.484059    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-dbus\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:44.484585 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.484102    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d856ebe3-2d7c-4833-9576-504511abb585-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.484585 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.484138    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-modprobe-d\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.484585 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.484270    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2218d4e3-f6ca-4473-8187-abc582397c27-etc-modprobe-d\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.488325 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.488301    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kfk8\" (UniqueName: \"kubernetes.io/projected/149b4a0e-4466-46a6-9e51-d2450702c4d0-kube-api-access-8kfk8\") pod \"node-resolver-n4zhm\" (UID: \"149b4a0e-4466-46a6-9e51-d2450702c4d0\") " pod="openshift-dns/node-resolver-n4zhm"
Sep 17 21:58:44.489465 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.489448    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:44.489553 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.489468    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:44.489553 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.489478    2567 projected.go:196] Error preparing data for projected volume kube-api-access-bwxxx for pod openshift-network-diagnostics/network-check-target-th2bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:44.489553 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.489545    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx podName:da365699-cc35-49b9-a86a-58abbcc127f2 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:44.989523237 +0000 UTC m=+1.998452772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bwxxx" (UniqueName: "kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx") pod "network-check-target-th2bd" (UID: "da365699-cc35-49b9-a86a-58abbcc127f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:44.490180 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.490166    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hsb9\" (UniqueName: \"kubernetes.io/projected/2218d4e3-f6ca-4473-8187-abc582397c27-kube-api-access-5hsb9\") pod \"tuned-v76w8\" (UID: \"2218d4e3-f6ca-4473-8187-abc582397c27\") " pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.492181 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.492161    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f58g\" (UniqueName: \"kubernetes.io/projected/0b2f2051-ca1f-4ba1-897c-7266d76556c4-kube-api-access-8f58g\") pod \"iptables-alerter-8k6br\" (UID: \"0b2f2051-ca1f-4ba1-897c-7266d76556c4\") " pod="openshift-network-operator/iptables-alerter-8k6br"
Sep 17 21:58:44.571954 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.571935    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:44.585272 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585250    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d856ebe3-2d7c-4833-9576-504511abb585-cnibin\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.585366 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585274    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d856ebe3-2d7c-4833-9576-504511abb585-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.585366 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585291    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-kubelet\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.585366 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585314    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-cni-bin\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.585366 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585337    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-cni-netd\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.585366 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585352    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-ovnkube-config\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.585630 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585368    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-device-dir\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.585630 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585378    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d856ebe3-2d7c-4833-9576-504511abb585-cnibin\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.585630 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585391    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-multus-socket-dir-parent\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.585630 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585378    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-kubelet\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.585630 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585415    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-cni-bin\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.585630 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585410    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-cni-netd\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.585630 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585483    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-multus-socket-dir-parent\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.585630 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585516    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-device-dir\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.585630 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585544    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-var-lib-kubelet\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.585630 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585564    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-var-lib-openvswitch\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.585630 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585589    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-var-lib-kubelet\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.585630 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585590    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-log-socket\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.585630 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585623    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.585630 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585629    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-var-lib-openvswitch\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585636    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-log-socket\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585646    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d856ebe3-2d7c-4833-9576-504511abb585-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585674    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585676    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-node-log\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585704    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-system-cni-dir\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585728    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-multus-cni-dir\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585745    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-system-cni-dir\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585707    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-node-log\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585757    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-var-lib-cni-bin\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585785    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-run-systemd\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585802    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-cnibin\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585812    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-multus-cni-dir\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585840    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d856ebe3-2d7c-4833-9576-504511abb585-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585852    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-cnibin\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585816    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-var-lib-cni-multus\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585860    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-var-lib-cni-bin\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585851    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-var-lib-cni-multus\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.586369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585879    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-run-systemd\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585895    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-run-multus-certs\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585921    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd82d87-0321-4c10-a183-6d6122eda0cc-host\") pod \"node-ca-xc5tg\" (UID: \"5dd82d87-0321-4c10-a183-6d6122eda0cc\") " pod="openshift-image-registry/node-ca-xc5tg"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585938    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-ovnkube-config\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585939    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-run-multus-certs\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585966    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5dd82d87-0321-4c10-a183-6d6122eda0cc-host\") pod \"node-ca-xc5tg\" (UID: \"5dd82d87-0321-4c10-a183-6d6122eda0cc\") " pod="openshift-image-registry/node-ca-xc5tg"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.585943    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpgmz\" (UniqueName: \"kubernetes.io/projected/5dd82d87-0321-4c10-a183-6d6122eda0cc-kube-api-access-bpgmz\") pod \"node-ca-xc5tg\" (UID: \"5dd82d87-0321-4c10-a183-6d6122eda0cc\") " pod="openshift-image-registry/node-ca-xc5tg"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586028    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-socket-dir\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586048    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-dbus\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586065    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d856ebe3-2d7c-4833-9576-504511abb585-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586069    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d856ebe3-2d7c-4833-9576-504511abb585-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586087    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-slash\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586109    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-etc-openvswitch\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586128    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-dbus\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586134    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586145    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-slash\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586165    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-524vk\" (UniqueName: \"kubernetes.io/projected/94979e3c-1ac8-4800-a60f-3d3552644bcc-kube-api-access-524vk\") pod \"network-metrics-daemon-fdxms\" (UID: \"94979e3c-1ac8-4800-a60f-3d3552644bcc\") " pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:44.587221 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586190    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-env-overrides\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586188    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586165    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-etc-openvswitch\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586220    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-etc-selinux\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586223    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-socket-dir\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586253    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd4fee5c-7168-471c-b29a-f92b77f59b67-multus-daemon-config\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586276    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-etc-kubernetes\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586300    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d856ebe3-2d7c-4833-9576-504511abb585-system-cni-dir\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586277    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-etc-selinux\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586330    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-run-netns\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586358    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-etc-kubernetes\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586360    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d856ebe3-2d7c-4833-9576-504511abb585-system-cni-dir\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586407    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-ovn-node-metrics-cert\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586418    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-run-netns\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586425    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5dd82d87-0321-4c10-a183-6d6122eda0cc-serviceca\") pod \"node-ca-xc5tg\" (UID: \"5dd82d87-0321-4c10-a183-6d6122eda0cc\") " pod="openshift-image-registry/node-ca-xc5tg"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586553    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586582    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-run-ovn\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.588033 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586609    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d856ebe3-2d7c-4833-9576-504511abb585-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586625    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d856ebe3-2d7c-4833-9576-504511abb585-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586631    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-env-overrides\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.586660    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586675    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-run-ovn\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.586714    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret podName:c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:45.086702811 +0000 UTC m=+2.095632346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret") pod "global-pull-secret-syncer-58crn" (UID: "c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586719    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-systemd-units\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586633    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-systemd-units\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586758    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-registration-dir\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586766    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd4fee5c-7168-471c-b29a-f92b77f59b67-multus-daemon-config\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586774    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d856ebe3-2d7c-4833-9576-504511abb585-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586782    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e8cae935-fc5f-46f9-ac7c-582f15caaf4c-agent-certs\") pod \"konnectivity-agent-qhmn8\" (UID: \"e8cae935-fc5f-46f9-ac7c-582f15caaf4c\") " pod="kube-system/konnectivity-agent-qhmn8"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586815    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-hostroot\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586830    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5dd82d87-0321-4c10-a183-6d6122eda0cc-serviceca\") pod \"node-ca-xc5tg\" (UID: \"5dd82d87-0321-4c10-a183-6d6122eda0cc\") " pod="openshift-image-registry/node-ca-xc5tg"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586857    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-hostroot\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586865    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs\") pod \"network-metrics-daemon-fdxms\" (UID: \"94979e3c-1ac8-4800-a60f-3d3552644bcc\") " pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586883    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-registration-dir\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.588605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586889    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzp59\" (UniqueName: \"kubernetes.io/projected/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-kube-api-access-dzp59\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586924    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfxrs\" (UniqueName: \"kubernetes.io/projected/2d5278a3-afa2-435f-ba85-6bd5658a483f-kube-api-access-gfxrs\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.586935    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586950    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-multus-conf-dir\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.586987    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs podName:94979e3c-1ac8-4800-a60f-3d3552644bcc nodeName:}" failed. No retries permitted until 2026-09-17 21:58:45.086972276 +0000 UTC m=+2.095901813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs") pod "network-metrics-daemon-fdxms" (UID: "94979e3c-1ac8-4800-a60f-3d3552644bcc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.586981    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcb7g\" (UniqueName: \"kubernetes.io/projected/bd4fee5c-7168-471c-b29a-f92b77f59b67-kube-api-access-wcb7g\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587060    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-multus-conf-dir\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587161    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-run-ovn-kubernetes\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587200    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-ovnkube-script-lib\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587227    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-sys-fs\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587244    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-host-run-ovn-kubernetes\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587252    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45tmj\" (UniqueName: \"kubernetes.io/projected/d856ebe3-2d7c-4833-9576-504511abb585-kube-api-access-45tmj\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587288    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-run-k8s-cni-cncf-io\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587311    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-run-netns\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587328    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2d5278a3-afa2-435f-ba85-6bd5658a483f-sys-fs\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587333    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-run-openvswitch\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587370    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-run-openvswitch\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.589153 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587414    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-run-k8s-cni-cncf-io\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.589699 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587420    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd4fee5c-7168-471c-b29a-f92b77f59b67-host-run-netns\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.589699 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587372    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e8cae935-fc5f-46f9-ac7c-582f15caaf4c-konnectivity-ca\") pod \"konnectivity-agent-qhmn8\" (UID: \"e8cae935-fc5f-46f9-ac7c-582f15caaf4c\") " pod="kube-system/konnectivity-agent-qhmn8"
Sep 17 21:58:44.589699 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587554    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd4fee5c-7168-471c-b29a-f92b77f59b67-cni-binary-copy\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.589699 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587581    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-kubelet-config\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:44.589699 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587645    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-ovnkube-script-lib\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.589699 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587661    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-kubelet-config\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:44.589699 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.587889    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e8cae935-fc5f-46f9-ac7c-582f15caaf4c-konnectivity-ca\") pod \"konnectivity-agent-qhmn8\" (UID: \"e8cae935-fc5f-46f9-ac7c-582f15caaf4c\") " pod="kube-system/konnectivity-agent-qhmn8"
Sep 17 21:58:44.589699 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.588126    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd4fee5c-7168-471c-b29a-f92b77f59b67-cni-binary-copy\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.589699 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.588717    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-ovn-node-metrics-cert\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.589699 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.588909    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e8cae935-fc5f-46f9-ac7c-582f15caaf4c-agent-certs\") pod \"konnectivity-agent-qhmn8\" (UID: \"e8cae935-fc5f-46f9-ac7c-582f15caaf4c\") " pod="kube-system/konnectivity-agent-qhmn8"
Sep 17 21:58:44.603160 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.603144    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcb7g\" (UniqueName: \"kubernetes.io/projected/bd4fee5c-7168-471c-b29a-f92b77f59b67-kube-api-access-wcb7g\") pod \"multus-q79d2\" (UID: \"bd4fee5c-7168-471c-b29a-f92b77f59b67\") " pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.607239 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.607222    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpgmz\" (UniqueName: \"kubernetes.io/projected/5dd82d87-0321-4c10-a183-6d6122eda0cc-kube-api-access-bpgmz\") pod \"node-ca-xc5tg\" (UID: \"5dd82d87-0321-4c10-a183-6d6122eda0cc\") " pod="openshift-image-registry/node-ca-xc5tg"
Sep 17 21:58:44.607319 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.607303    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-524vk\" (UniqueName: \"kubernetes.io/projected/94979e3c-1ac8-4800-a60f-3d3552644bcc-kube-api-access-524vk\") pod \"network-metrics-daemon-fdxms\" (UID: \"94979e3c-1ac8-4800-a60f-3d3552644bcc\") " pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:44.611423 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.611397    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfxrs\" (UniqueName: \"kubernetes.io/projected/2d5278a3-afa2-435f-ba85-6bd5658a483f-kube-api-access-gfxrs\") pod \"aws-ebs-csi-driver-node-hfsng\" (UID: \"2d5278a3-afa2-435f-ba85-6bd5658a483f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.611564 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.611543    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzp59\" (UniqueName: \"kubernetes.io/projected/50ab8fb7-7f5b-401f-a16b-740d063b3ee6-kube-api-access-dzp59\") pod \"ovnkube-node-2kp5t\" (UID: \"50ab8fb7-7f5b-401f-a16b-740d063b3ee6\") " pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.611661 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.611647    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45tmj\" (UniqueName: \"kubernetes.io/projected/d856ebe3-2d7c-4833-9576-504511abb585-kube-api-access-45tmj\") pod \"multus-additional-cni-plugins-q9nmn\" (UID: \"d856ebe3-2d7c-4833-9576-504511abb585\") " pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.621594 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.621309    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:44.693920 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.693860    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v76w8"
Sep 17 21:58:44.708357 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.708333    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q79d2"
Sep 17 21:58:44.726901 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.726881    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n4zhm"
Sep 17 21:58:44.741549 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.741531    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8k6br"
Sep 17 21:58:44.753525 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.753505    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:58:44.758275 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.758261    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhmn8"
Sep 17 21:58:44.762602 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:44.762576    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a12a94a946441191f85a154c16cabf.slice/crio-a9c6193d79c2dba83ee1c346a8fb727e8ee8ce35cfc73c11ddf1bd28bb47f583 WatchSource:0}: Error finding container a9c6193d79c2dba83ee1c346a8fb727e8ee8ce35cfc73c11ddf1bd28bb47f583: Status 404 returned error can't find the container with id a9c6193d79c2dba83ee1c346a8fb727e8ee8ce35cfc73c11ddf1bd28bb47f583
Sep 17 21:58:44.763076 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:44.763058    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149b4a0e_4466_46a6_9e51_d2450702c4d0.slice/crio-00c9805d27dea75aabb9579dc255fa889131b504470e2b2a420dc765525d7ba0 WatchSource:0}: Error finding container 00c9805d27dea75aabb9579dc255fa889131b504470e2b2a420dc765525d7ba0: Status 404 returned error can't find the container with id 00c9805d27dea75aabb9579dc255fa889131b504470e2b2a420dc765525d7ba0
Sep 17 21:58:44.763527 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:44.763508    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4fee5c_7168_471c_b29a_f92b77f59b67.slice/crio-fd129fedd905fdd07c8c3b1a7f6b7fc861436da2e61d8cb14bb9028b746e6c23 WatchSource:0}: Error finding container fd129fedd905fdd07c8c3b1a7f6b7fc861436da2e61d8cb14bb9028b746e6c23: Status 404 returned error can't find the container with id fd129fedd905fdd07c8c3b1a7f6b7fc861436da2e61d8cb14bb9028b746e6c23
Sep 17 21:58:44.764719 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:44.764697    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19258485fc6925eb879b1d7211e2fc7.slice/crio-ad1b3e58318272831372fb06a881099380569e49df8eb2159fe3226d13a9278d WatchSource:0}: Error finding container ad1b3e58318272831372fb06a881099380569e49df8eb2159fe3226d13a9278d: Status 404 returned error can't find the container with id ad1b3e58318272831372fb06a881099380569e49df8eb2159fe3226d13a9278d
Sep 17 21:58:44.766294 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:44.766276    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2f2051_ca1f_4ba1_897c_7266d76556c4.slice/crio-8a796dd91b4a96debb396634ae641ce4f56cf2a9e33c4c7f0aa95dd7efb61a5f WatchSource:0}: Error finding container 8a796dd91b4a96debb396634ae641ce4f56cf2a9e33c4c7f0aa95dd7efb61a5f: Status 404 returned error can't find the container with id 8a796dd91b4a96debb396634ae641ce4f56cf2a9e33c4c7f0aa95dd7efb61a5f
Sep 17 21:58:44.766836 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:44.766767    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ab8fb7_7f5b_401f_a16b_740d063b3ee6.slice/crio-d815255d8ec446bbe7055cc0b7da36963873307c15e1d0e703cf3af6d84940f8 WatchSource:0}: Error finding container d815255d8ec446bbe7055cc0b7da36963873307c15e1d0e703cf3af6d84940f8: Status 404 returned error can't find the container with id d815255d8ec446bbe7055cc0b7da36963873307c15e1d0e703cf3af6d84940f8
Sep 17 21:58:44.766958 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.766938    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng"
Sep 17 21:58:44.769584 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.769546    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:58:44.773341 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:44.773249    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cae935_fc5f_46f9_ac7c_582f15caaf4c.slice/crio-aef1f1efc67d5ac73e101ee046bde801d35a6c061c9e80c0b6035250f53bf62d WatchSource:0}: Error finding container aef1f1efc67d5ac73e101ee046bde801d35a6c061c9e80c0b6035250f53bf62d: Status 404 returned error can't find the container with id aef1f1efc67d5ac73e101ee046bde801d35a6c061c9e80c0b6035250f53bf62d
Sep 17 21:58:44.780593 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:44.780572    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5278a3_afa2_435f_ba85_6bd5658a483f.slice/crio-7b36e2c32485ab86ea10c429a6f38c9c863fcf13c6351258803a1fcb8882de44 WatchSource:0}: Error finding container 7b36e2c32485ab86ea10c429a6f38c9c863fcf13c6351258803a1fcb8882de44: Status 404 returned error can't find the container with id 7b36e2c32485ab86ea10c429a6f38c9c863fcf13c6351258803a1fcb8882de44
Sep 17 21:58:44.800774 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.800759    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xc5tg"
Sep 17 21:58:44.801849 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.801835    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q9nmn"
Sep 17 21:58:44.810322 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:44.810303    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd82d87_0321_4c10_a183_6d6122eda0cc.slice/crio-6dce177cc006e05cc82bdc63101f98e02ec26b200021a34ef010fc3ac39fcda6 WatchSource:0}: Error finding container 6dce177cc006e05cc82bdc63101f98e02ec26b200021a34ef010fc3ac39fcda6: Status 404 returned error can't find the container with id 6dce177cc006e05cc82bdc63101f98e02ec26b200021a34ef010fc3ac39fcda6
Sep 17 21:58:44.813544 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:44.813526    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd856ebe3_2d7c_4833_9576_504511abb585.slice/crio-3a098ea106400802461f81bebb5a3037b9bd37c93bfe5f63af3235cd3c9128f0 WatchSource:0}: Error finding container 3a098ea106400802461f81bebb5a3037b9bd37c93bfe5f63af3235cd3c9128f0: Status 404 returned error can't find the container with id 3a098ea106400802461f81bebb5a3037b9bd37c93bfe5f63af3235cd3c9128f0
Sep 17 21:58:44.990507 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:44.990424    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxxx\" (UniqueName: \"kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx\") pod \"network-check-target-th2bd\" (UID: \"da365699-cc35-49b9-a86a-58abbcc127f2\") " pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:58:44.990610 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.990584    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:44.990610 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.990599    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:44.990610 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.990608    2567 projected.go:196] Error preparing data for projected volume kube-api-access-bwxxx for pod openshift-network-diagnostics/network-check-target-th2bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:44.990715 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:44.990648    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx podName:da365699-cc35-49b9-a86a-58abbcc127f2 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:45.990636512 +0000 UTC m=+2.999566037 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwxxx" (UniqueName: "kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx") pod "network-check-target-th2bd" (UID: "da365699-cc35-49b9-a86a-58abbcc127f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:45.090829 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.090804    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:45.090961 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.090837    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs\") pod \"network-metrics-daemon-fdxms\" (UID: \"94979e3c-1ac8-4800-a60f-3d3552644bcc\") " pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:45.090961 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:45.090929    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:45.090961 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:45.090949    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:45.091105 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:45.090971    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs podName:94979e3c-1ac8-4800-a60f-3d3552644bcc nodeName:}" failed. No retries permitted until 2026-09-17 21:58:46.090959166 +0000 UTC m=+3.099888697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs") pod "network-metrics-daemon-fdxms" (UID: "94979e3c-1ac8-4800-a60f-3d3552644bcc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:45.091105 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:45.091019    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret podName:c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:46.090988162 +0000 UTC m=+3.099917698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret") pod "global-pull-secret-syncer-58crn" (UID: "c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:45.142733 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:58:45.142710    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2218d4e3_f6ca_4473_8187_abc582397c27.slice/crio-4fcc7a2f033da0f61b86f4c5ecaa7b4a3b411c69ff8157a64a2e65d1b406ec67 WatchSource:0}: Error finding container 4fcc7a2f033da0f61b86f4c5ecaa7b4a3b411c69ff8157a64a2e65d1b406ec67: Status 404 returned error can't find the container with id 4fcc7a2f033da0f61b86f4c5ecaa7b4a3b411c69ff8157a64a2e65d1b406ec67
Sep 17 21:58:45.422133 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.422018    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:53:44 +0000 UTC" deadline="2028-05-28 10:10:42.328548454 +0000 UTC"
Sep 17 21:58:45.422133 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.422079    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14844h11m56.906474377s"
Sep 17 21:58:45.660914 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.660858    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9nmn" event={"ID":"d856ebe3-2d7c-4833-9576-504511abb585","Type":"ContainerStarted","Data":"3a098ea106400802461f81bebb5a3037b9bd37c93bfe5f63af3235cd3c9128f0"}
Sep 17 21:58:45.679499 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.679428    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xc5tg" event={"ID":"5dd82d87-0321-4c10-a183-6d6122eda0cc","Type":"ContainerStarted","Data":"6dce177cc006e05cc82bdc63101f98e02ec26b200021a34ef010fc3ac39fcda6"}
Sep 17 21:58:45.684506 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.684262    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhmn8" event={"ID":"e8cae935-fc5f-46f9-ac7c-582f15caaf4c","Type":"ContainerStarted","Data":"aef1f1efc67d5ac73e101ee046bde801d35a6c061c9e80c0b6035250f53bf62d"}
Sep 17 21:58:45.687930 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.687906    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:45.696713 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.696683    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal" event={"ID":"d19258485fc6925eb879b1d7211e2fc7","Type":"ContainerStarted","Data":"ad1b3e58318272831372fb06a881099380569e49df8eb2159fe3226d13a9278d"}
Sep 17 21:58:45.714111 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.714086    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal" event={"ID":"a6a12a94a946441191f85a154c16cabf","Type":"ContainerStarted","Data":"a9c6193d79c2dba83ee1c346a8fb727e8ee8ce35cfc73c11ddf1bd28bb47f583"}
Sep 17 21:58:45.729076 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.728866    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v76w8" event={"ID":"2218d4e3-f6ca-4473-8187-abc582397c27","Type":"ContainerStarted","Data":"4fcc7a2f033da0f61b86f4c5ecaa7b4a3b411c69ff8157a64a2e65d1b406ec67"}
Sep 17 21:58:45.756251 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.756208    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng" event={"ID":"2d5278a3-afa2-435f-ba85-6bd5658a483f","Type":"ContainerStarted","Data":"7b36e2c32485ab86ea10c429a6f38c9c863fcf13c6351258803a1fcb8882de44"}
Sep 17 21:58:45.777694 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.777661    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t" event={"ID":"50ab8fb7-7f5b-401f-a16b-740d063b3ee6","Type":"ContainerStarted","Data":"d815255d8ec446bbe7055cc0b7da36963873307c15e1d0e703cf3af6d84940f8"}
Sep 17 21:58:45.791772 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.791706    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8k6br" event={"ID":"0b2f2051-ca1f-4ba1-897c-7266d76556c4","Type":"ContainerStarted","Data":"8a796dd91b4a96debb396634ae641ce4f56cf2a9e33c4c7f0aa95dd7efb61a5f"}
Sep 17 21:58:45.799251 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.799192    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q79d2" event={"ID":"bd4fee5c-7168-471c-b29a-f92b77f59b67","Type":"ContainerStarted","Data":"fd129fedd905fdd07c8c3b1a7f6b7fc861436da2e61d8cb14bb9028b746e6c23"}
Sep 17 21:58:45.826903 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.826871    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n4zhm" event={"ID":"149b4a0e-4466-46a6-9e51-d2450702c4d0","Type":"ContainerStarted","Data":"00c9805d27dea75aabb9579dc255fa889131b504470e2b2a420dc765525d7ba0"}
Sep 17 21:58:45.997754 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:45.997147    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxxx\" (UniqueName: \"kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx\") pod \"network-check-target-th2bd\" (UID: \"da365699-cc35-49b9-a86a-58abbcc127f2\") " pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:58:45.997754 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:45.997312    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:45.997754 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:45.997334    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:45.997754 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:45.997346    2567 projected.go:196] Error preparing data for projected volume kube-api-access-bwxxx for pod openshift-network-diagnostics/network-check-target-th2bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:45.997754 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:45.997399    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx podName:da365699-cc35-49b9-a86a-58abbcc127f2 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:47.997381968 +0000 UTC m=+5.006311503 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwxxx" (UniqueName: "kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx") pod "network-check-target-th2bd" (UID: "da365699-cc35-49b9-a86a-58abbcc127f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:46.098623 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:46.097821    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs\") pod \"network-metrics-daemon-fdxms\" (UID: \"94979e3c-1ac8-4800-a60f-3d3552644bcc\") " pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:46.098623 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:46.097908    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:46.098623 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:46.098108    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:46.098623 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:46.098171    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret podName:c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:48.098153239 +0000 UTC m=+5.107082776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret") pod "global-pull-secret-syncer-58crn" (UID: "c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:46.098623 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:46.098542    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:46.098623 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:46.098591    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs podName:94979e3c-1ac8-4800-a60f-3d3552644bcc nodeName:}" failed. No retries permitted until 2026-09-17 21:58:48.098576487 +0000 UTC m=+5.107506010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs") pod "network-metrics-daemon-fdxms" (UID: "94979e3c-1ac8-4800-a60f-3d3552644bcc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:46.423638 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:46.423007    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:53:44 +0000 UTC" deadline="2028-04-15 12:23:44.694722854 +0000 UTC"
Sep 17 21:58:46.423638 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:46.423046    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13814h24m58.271681965s"
Sep 17 21:58:46.635509 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:46.635475    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:58:46.635682 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:46.635593    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:58:46.635990 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:46.635969    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:46.636121 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:46.636075    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:58:46.636191 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:46.636141    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:46.636242 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:46.636226    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:58:46.842047 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:46.841953    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:48.014544 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:48.014514    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxxx\" (UniqueName: \"kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx\") pod \"network-check-target-th2bd\" (UID: \"da365699-cc35-49b9-a86a-58abbcc127f2\") " pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:58:48.015047 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:48.014709    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:48.015047 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:48.014727    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:48.015047 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:48.014738    2567 projected.go:196] Error preparing data for projected volume kube-api-access-bwxxx for pod openshift-network-diagnostics/network-check-target-th2bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:48.015047 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:48.014790    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx podName:da365699-cc35-49b9-a86a-58abbcc127f2 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:52.014773527 +0000 UTC m=+9.023703063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwxxx" (UniqueName: "kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx") pod "network-check-target-th2bd" (UID: "da365699-cc35-49b9-a86a-58abbcc127f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:48.115907 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:48.115150    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs\") pod \"network-metrics-daemon-fdxms\" (UID: \"94979e3c-1ac8-4800-a60f-3d3552644bcc\") " pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:48.115907 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:48.115249    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:48.115907 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:48.115366    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:48.115907 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:48.115421    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret podName:c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:52.115404067 +0000 UTC m=+9.124333607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret") pod "global-pull-secret-syncer-58crn" (UID: "c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:48.115907 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:48.115818    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:48.115907 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:48.115877    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs podName:94979e3c-1ac8-4800-a60f-3d3552644bcc nodeName:}" failed. No retries permitted until 2026-09-17 21:58:52.115863583 +0000 UTC m=+9.124793118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs") pod "network-metrics-daemon-fdxms" (UID: "94979e3c-1ac8-4800-a60f-3d3552644bcc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:48.634774 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:48.634707    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:48.634929 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:48.634744    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:48.634929 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:48.634892    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:58:48.635016 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:48.634983    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:58:48.635074 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:48.634744    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:58:48.635125 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:48.635088    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:58:50.635538 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:50.635016    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:58:50.635538 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:50.635154    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:58:50.635538 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:50.635368    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:50.635538 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:50.635484    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:58:50.636250 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:50.635555    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:50.636250 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:50.635623    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:58:52.048504 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:52.048477    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxxx\" (UniqueName: \"kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx\") pod \"network-check-target-th2bd\" (UID: \"da365699-cc35-49b9-a86a-58abbcc127f2\") " pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:58:52.048878 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:52.048645    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:52.048878 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:52.048664    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:52.048878 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:52.048675    2567 projected.go:196] Error preparing data for projected volume kube-api-access-bwxxx for pod openshift-network-diagnostics/network-check-target-th2bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:52.048878 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:52.048733    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx podName:da365699-cc35-49b9-a86a-58abbcc127f2 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:00.048713734 +0000 UTC m=+17.057643276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwxxx" (UniqueName: "kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx") pod "network-check-target-th2bd" (UID: "da365699-cc35-49b9-a86a-58abbcc127f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:52.149105 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:52.149068    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:52.149304 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:52.149126    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs\") pod \"network-metrics-daemon-fdxms\" (UID: \"94979e3c-1ac8-4800-a60f-3d3552644bcc\") " pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:52.149304 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:52.149244    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:52.149304 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:52.149273    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:52.149460 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:52.149309    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret podName:c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:00.149290668 +0000 UTC m=+17.158220207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret") pod "global-pull-secret-syncer-58crn" (UID: "c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:52.149460 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:52.149328    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs podName:94979e3c-1ac8-4800-a60f-3d3552644bcc nodeName:}" failed. No retries permitted until 2026-09-17 21:59:00.149318182 +0000 UTC m=+17.158247714 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs") pod "network-metrics-daemon-fdxms" (UID: "94979e3c-1ac8-4800-a60f-3d3552644bcc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:52.635415 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:52.635381    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:52.635604 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:52.635425    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:58:52.635604 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:52.635425    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:52.635604 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:52.635527    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:58:52.635772 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:52.635644    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:58:52.635772 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:52.635730    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:58:54.634919 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:54.634884    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:54.635477 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:54.634925    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:54.635477 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:54.634884    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:58:54.635477 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:54.635031    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:58:54.635477 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:54.635100    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:58:54.635477 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:54.635158    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:58:56.635038 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:56.634988    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:56.635038 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:56.635025    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:58:56.635528 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:56.635030    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:56.635528 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:56.635134    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:58:56.635528 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:56.635268    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:58:56.635528 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:56.635385    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:58:58.635126 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:58.635091    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:58:58.635518 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:58.635207    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:58:58.635518 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:58.635091    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:58:58.635518 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:58:58.635091    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:58:58.635518 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:58.635283    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:58:58.635518 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:58:58.635333    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:59:00.103380 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:00.103347    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxxx\" (UniqueName: \"kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx\") pod \"network-check-target-th2bd\" (UID: \"da365699-cc35-49b9-a86a-58abbcc127f2\") " pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:00.103818 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:00.103507    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:59:00.103818 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:00.103539    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:59:00.103818 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:00.103551    2567 projected.go:196] Error preparing data for projected volume kube-api-access-bwxxx for pod openshift-network-diagnostics/network-check-target-th2bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:59:00.103818 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:00.103608    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx podName:da365699-cc35-49b9-a86a-58abbcc127f2 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:16.103590415 +0000 UTC m=+33.112519937 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwxxx" (UniqueName: "kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx") pod "network-check-target-th2bd" (UID: "da365699-cc35-49b9-a86a-58abbcc127f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:59:00.203797 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:00.203753    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:00.203797 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:00.203801    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs\") pod \"network-metrics-daemon-fdxms\" (UID: \"94979e3c-1ac8-4800-a60f-3d3552644bcc\") " pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:00.204024 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:00.203918    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:59:00.204024 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:00.203937    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:59:00.204024 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:00.203991    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret podName:c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:16.203968122 +0000 UTC m=+33.212897663 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret") pod "global-pull-secret-syncer-58crn" (UID: "c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:59:00.204024 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:00.204024    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs podName:94979e3c-1ac8-4800-a60f-3d3552644bcc nodeName:}" failed. No retries permitted until 2026-09-17 21:59:16.204013425 +0000 UTC m=+33.212942948 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs") pod "network-metrics-daemon-fdxms" (UID: "94979e3c-1ac8-4800-a60f-3d3552644bcc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:59:00.635171 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:00.635142    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:00.635171 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:00.635157    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:00.635410 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:00.635145    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:00.635410 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:00.635251    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:59:00.635410 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:00.635348    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:59:00.635532 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:00.635422    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:59:02.634831 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:02.634793    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:02.635461 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:02.634845    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:02.635461 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:02.634793    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:02.635461 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:02.634919    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:59:02.635461 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:02.635074    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:59:02.635461 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:02.635158    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:59:03.869599 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:03.869116    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal" event={"ID":"a6a12a94a946441191f85a154c16cabf","Type":"ContainerStarted","Data":"d43c370f7591f17228461a36a6518da691b7fd460dc7f4469f37a45c459c03fb"}
Sep 17 21:59:03.870881 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:03.870854    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v76w8" event={"ID":"2218d4e3-f6ca-4473-8187-abc582397c27","Type":"ContainerStarted","Data":"7479c92ec1d1a57c3a6cd225b2730ac70ad300db8239b3f9400918bf5572b056"}
Sep 17 21:59:03.873261 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:03.873239    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t" event={"ID":"50ab8fb7-7f5b-401f-a16b-740d063b3ee6","Type":"ContainerStarted","Data":"d273636148cde9ad3c7176a60cef0e3489b2a748c13c1f9bac9632ed34bcc85f"}
Sep 17 21:59:03.873353 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:03.873300    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t" event={"ID":"50ab8fb7-7f5b-401f-a16b-740d063b3ee6","Type":"ContainerStarted","Data":"06fc3bbcd5465cb805fb74326fccd1b64b621ce58942b5f545809c27010bf409"}
Sep 17 21:59:03.873353 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:03.873330    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t" event={"ID":"50ab8fb7-7f5b-401f-a16b-740d063b3ee6","Type":"ContainerStarted","Data":"11fc02a6f3eb4f5c12e5fdcb03067d5c86d78cddf5e8d04375ef57a7f766f27d"}
Sep 17 21:59:03.874981 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:03.874955    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q79d2" event={"ID":"bd4fee5c-7168-471c-b29a-f92b77f59b67","Type":"ContainerStarted","Data":"6c7aa68ad4f1cd13a464b40520ee169ee7d883c2dd0d66329f3caab495e8f018"}
Sep 17 21:59:03.898341 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:03.898281    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v76w8" podStartSLOduration=2.893390714 podStartE2EDuration="20.898262043s" totalImagesPullingTime="18.004871329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:03.896421882 +0000 UTC m=+20.905351418" watchObservedRunningTime="2026-09-17 21:59:03.898262043 +0000 UTC m=+20.907191585"
Sep 17 21:59:03.913534 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:03.913490    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-q79d2" podStartSLOduration=2.288956242 podStartE2EDuration="20.913474576s" totalImagesPullingTime="18.624518334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:03.913064439 +0000 UTC m=+20.921993979" watchObservedRunningTime="2026-09-17 21:59:03.913474576 +0000 UTC m=+20.922404115"
Sep 17 21:59:04.635188 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.635153    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:04.635412 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:04.635296    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:59:04.635412 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.635313    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:04.635542 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:04.635445    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:59:04.635542 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.635469    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:04.635646 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:04.635551    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:59:04.879373 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.879344    2567 generic.go:356] "Generic (PLEG): container finished" podID="d856ebe3-2d7c-4833-9576-504511abb585" containerID="ce27ca1e228d50cf3d3745ea8ad99fd7feab16c6a992ce6edfce8549435eefd9" exitCode=0
Sep 17 21:59:04.880287 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.879410    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9nmn" event={"ID":"d856ebe3-2d7c-4833-9576-504511abb585","Type":"ContainerDied","Data":"ce27ca1e228d50cf3d3745ea8ad99fd7feab16c6a992ce6edfce8549435eefd9"}
Sep 17 21:59:04.881102 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.881022    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xc5tg" event={"ID":"5dd82d87-0321-4c10-a183-6d6122eda0cc","Type":"ContainerStarted","Data":"430edaa6d9aa68edd96ac873b54c934a84edc39fac6518b129701717c3911e78"}
Sep 17 21:59:04.884954 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.884914    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhmn8" event={"ID":"e8cae935-fc5f-46f9-ac7c-582f15caaf4c","Type":"ContainerStarted","Data":"1276cfeeac6f4860eff1080c27d35ee4de873cef63c469fe7aae1df9b98839ab"}
Sep 17 21:59:04.886374 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.886322    2567 generic.go:356] "Generic (PLEG): container finished" podID="d19258485fc6925eb879b1d7211e2fc7" containerID="7e98ddde9defda53856a6a237f08174373027fd6d69caa821e4be59f849b2175" exitCode=0
Sep 17 21:59:04.886467 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.886411    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal" event={"ID":"d19258485fc6925eb879b1d7211e2fc7","Type":"ContainerDied","Data":"7e98ddde9defda53856a6a237f08174373027fd6d69caa821e4be59f849b2175"}
Sep 17 21:59:04.888205 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.888187    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng" event={"ID":"2d5278a3-afa2-435f-ba85-6bd5658a483f","Type":"ContainerStarted","Data":"98e127ff4ddd142a22a9ceccc73f795948ab663baed0ef6e47bb0fa24e4c3992"}
Sep 17 21:59:04.890715 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.890698    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t" event={"ID":"50ab8fb7-7f5b-401f-a16b-740d063b3ee6","Type":"ContainerStarted","Data":"d57984c8d0ba08b1597f040153f2880f4125d37eff959d8872cf76564a915705"}
Sep 17 21:59:04.890807 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.890748    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t" event={"ID":"50ab8fb7-7f5b-401f-a16b-740d063b3ee6","Type":"ContainerStarted","Data":"5c631b27a3daebe661ffbe8669d0a44d8801139cc0c4f13d722ea8e2b185de7d"}
Sep 17 21:59:04.890807 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.890776    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t" event={"ID":"50ab8fb7-7f5b-401f-a16b-740d063b3ee6","Type":"ContainerStarted","Data":"d00474496ed61e72e7abc186a1597c01b539d118df7ac7d0d92d5898b04d8c9e"}
Sep 17 21:59:04.891960 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.891941    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8k6br" event={"ID":"0b2f2051-ca1f-4ba1-897c-7266d76556c4","Type":"ContainerStarted","Data":"a9d85c2438faa8ff45b73aa6f109ac4b43ca2556b76a82474d7708a6d661a912"}
Sep 17 21:59:04.893300 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.893279    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n4zhm" event={"ID":"149b4a0e-4466-46a6-9e51-d2450702c4d0","Type":"ContainerStarted","Data":"dfe8bc9c5f9544878c9f41d5bd0932ff5f10a37c5c5281cbe78096082ba25e7b"}
Sep 17 21:59:04.899751 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.899707    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-197.ec2.internal" podStartSLOduration=20.899693756 podStartE2EDuration="20.899693756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:03.926579271 +0000 UTC m=+20.935508814" watchObservedRunningTime="2026-09-17 21:59:04.899693756 +0000 UTC m=+21.908623303"
Sep 17 21:59:04.925013 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.924953    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n4zhm" podStartSLOduration=3.58032501 podStartE2EDuration="21.924939743s" totalImagesPullingTime="18.344614733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:04.912393023 +0000 UTC m=+21.921322563" watchObservedRunningTime="2026-09-17 21:59:04.924939743 +0000 UTC m=+21.933869275"
Sep 17 21:59:04.925160 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.925060    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8k6br" podStartSLOduration=3.580653452 podStartE2EDuration="21.925053995s" totalImagesPullingTime="18.344400543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:04.925018319 +0000 UTC m=+21.933947858" watchObservedRunningTime="2026-09-17 21:59:04.925053995 +0000 UTC m=+21.933983535"
Sep 17 21:59:04.939027 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.938975    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qhmn8" podStartSLOduration=7.64116279 podStartE2EDuration="21.938964138s" totalImagesPullingTime="14.297801348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:04.938631716 +0000 UTC m=+21.947561256" watchObservedRunningTime="2026-09-17 21:59:04.938964138 +0000 UTC m=+21.947893679"
Sep 17 21:59:04.952898 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:04.952861    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xc5tg" podStartSLOduration=3.650261732 podStartE2EDuration="21.952849434s" totalImagesPullingTime="18.302587702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:04.952732375 +0000 UTC m=+21.961661918" watchObservedRunningTime="2026-09-17 21:59:04.952849434 +0000 UTC m=+21.961778975"
Sep 17 21:59:05.347080 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:05.346857    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 21:59:05.432057 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:05.431945    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T21:59:05.347076737Z","UUID":"dfbcfe97-d8c3-42f9-9f8f-31a3ec698a10","Handler":null,"Name":"","Endpoint":""}
Sep 17 21:59:05.434141 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:05.434118    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 21:59:05.434269 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:05.434150    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 21:59:05.898620 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:05.898559    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal" event={"ID":"d19258485fc6925eb879b1d7211e2fc7","Type":"ContainerStarted","Data":"feb5b41b46da4de7f94e4d287a6d823bacd02037c8a00e656163d36258545dab"}
Sep 17 21:59:05.902022 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:05.901329    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng" event={"ID":"2d5278a3-afa2-435f-ba85-6bd5658a483f","Type":"ContainerStarted","Data":"7661eabcfdf8190a20fc2c153979c3e8d50f1f82319828352061a87f31f9de84"}
Sep 17 21:59:06.634853 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:06.634824    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:06.634853 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:06.634842    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:06.635052 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:06.634940    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:59:06.635052 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:06.634954    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:06.635153 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:06.635068    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:59:06.635209 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:06.635152    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:59:06.907677 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:06.907597    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng" event={"ID":"2d5278a3-afa2-435f-ba85-6bd5658a483f","Type":"ContainerStarted","Data":"4ff0612131e0307bd7b9eba2ae7f89ee0551a9a9bb49c31390d5bf35aebe7703"}
Sep 17 21:59:06.910926 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:06.910894    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t" event={"ID":"50ab8fb7-7f5b-401f-a16b-740d063b3ee6","Type":"ContainerStarted","Data":"d5b5c2f94561f9c5ff6f03fedcfcd3fb70196c25e3f56ac3b3b32d0fa4714a10"}
Sep 17 21:59:06.924203 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:06.924164    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hfsng" podStartSLOduration=3.079870258 podStartE2EDuration="23.924143939s" totalImagesPullingTime="20.844273681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:06.924094374 +0000 UTC m=+23.933023915" watchObservedRunningTime="2026-09-17 21:59:06.924143939 +0000 UTC m=+23.933073458"
Sep 17 21:59:06.924743 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:06.924722    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-197.ec2.internal" podStartSLOduration=22.924714216 podStartE2EDuration="22.924714216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:05.915264429 +0000 UTC m=+22.924193970" watchObservedRunningTime="2026-09-17 21:59:06.924714216 +0000 UTC m=+23.933643756"
Sep 17 21:59:07.384568 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:07.384533    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qhmn8"
Sep 17 21:59:07.385178 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:07.385159    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qhmn8"
Sep 17 21:59:07.914114 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:07.913919    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qhmn8"
Sep 17 21:59:07.914609 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:07.914302    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qhmn8"
Sep 17 21:59:08.635214 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:08.635137    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:08.635214 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:08.635167    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:08.635428 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:08.635235    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:08.635428 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:08.635249    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:59:08.635428 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:08.635329    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:59:08.635428 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:08.635395    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:59:09.923478 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:09.923284    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t" event={"ID":"50ab8fb7-7f5b-401f-a16b-740d063b3ee6","Type":"ContainerStarted","Data":"8f583d389a3bc5008f5d349e737fb27cc13ce8f7b2a6c553166638dffa05970b"}
Sep 17 21:59:09.923893 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:09.923535    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:59:09.925133 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:09.925103    2567 generic.go:356] "Generic (PLEG): container finished" podID="d856ebe3-2d7c-4833-9576-504511abb585" containerID="82902ef5d1458bbc747a3b325c2f061f6b3eb707191009be2785e1726bd5a8a2" exitCode=0
Sep 17 21:59:09.925258 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:09.925180    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9nmn" event={"ID":"d856ebe3-2d7c-4833-9576-504511abb585","Type":"ContainerDied","Data":"82902ef5d1458bbc747a3b325c2f061f6b3eb707191009be2785e1726bd5a8a2"}
Sep 17 21:59:09.939184 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:09.939165    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:59:09.955286 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:09.955249    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t" podStartSLOduration=8.114024752 podStartE2EDuration="26.955237092s" totalImagesPullingTime="18.84121234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:09.953829863 +0000 UTC m=+26.962759415" watchObservedRunningTime="2026-09-17 21:59:09.955237092 +0000 UTC m=+26.964166633"
Sep 17 21:59:10.634605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:10.634583    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:10.634691 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:10.634583    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:10.634737 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:10.634696    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:59:10.634795 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:10.634584    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:10.634795 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:10.634752    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:59:10.634884 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:10.634815    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:59:10.930066 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:10.929974    2567 generic.go:356] "Generic (PLEG): container finished" podID="d856ebe3-2d7c-4833-9576-504511abb585" containerID="0d5fe4f852a9bb70b691eeb974a67ee89e7a377efe8472bd1eceee52195bf03a" exitCode=0
Sep 17 21:59:10.930066 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:10.930044    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9nmn" event={"ID":"d856ebe3-2d7c-4833-9576-504511abb585","Type":"ContainerDied","Data":"0d5fe4f852a9bb70b691eeb974a67ee89e7a377efe8472bd1eceee52195bf03a"}
Sep 17 21:59:10.930609 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:10.930194    2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 21:59:10.930767 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:10.930742    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:59:10.945703 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:10.945684    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:59:11.420537 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:11.419897    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-58crn"]
Sep 17 21:59:11.420537 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:11.420043    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:11.420537 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:11.420444    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:59:11.425076 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:11.423638    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fdxms"]
Sep 17 21:59:11.425076 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:11.423802    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:11.425076 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:11.423954    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:59:11.425076 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:11.424262    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-th2bd"]
Sep 17 21:59:11.425076 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:11.424364    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:11.425076 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:11.424467    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:59:11.933989 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:11.933961    2567 generic.go:356] "Generic (PLEG): container finished" podID="d856ebe3-2d7c-4833-9576-504511abb585" containerID="63ec2c217a0dd4fd6a127c4dc55a524303df62af9ba8c7e8b608518cf0c1c9e9" exitCode=0
Sep 17 21:59:11.934460 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:11.934039    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9nmn" event={"ID":"d856ebe3-2d7c-4833-9576-504511abb585","Type":"ContainerDied","Data":"63ec2c217a0dd4fd6a127c4dc55a524303df62af9ba8c7e8b608518cf0c1c9e9"}
Sep 17 21:59:11.934460 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:11.934109    2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 21:59:12.635376 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:12.635333    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:12.635376 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:12.635359    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:12.635578 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:12.635450    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:59:12.635578 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:12.635549    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:59:12.699675 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:12.699646    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:59:13.636130 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:13.636103    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:13.636481 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:13.636226    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:59:14.634786 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:14.634751    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:14.634937 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:14.634752    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:14.634937 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:14.634852    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:59:14.635069 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:14.634992    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:59:15.634878 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:15.634843    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:15.635309 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:15.634987    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fdxms" podUID="94979e3c-1ac8-4800-a60f-3d3552644bcc"
Sep 17 21:59:16.123932 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:16.123895    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxxx\" (UniqueName: \"kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx\") pod \"network-check-target-th2bd\" (UID: \"da365699-cc35-49b9-a86a-58abbcc127f2\") " pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:16.124126 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:16.124048    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:59:16.124126 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:16.124062    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:59:16.124126 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:16.124070    2567 projected.go:196] Error preparing data for projected volume kube-api-access-bwxxx for pod openshift-network-diagnostics/network-check-target-th2bd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:59:16.124126 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:16.124123    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx podName:da365699-cc35-49b9-a86a-58abbcc127f2 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:48.124106988 +0000 UTC m=+65.133036530 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwxxx" (UniqueName: "kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx") pod "network-check-target-th2bd" (UID: "da365699-cc35-49b9-a86a-58abbcc127f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:59:16.224537 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:16.224503    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:16.224537 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:16.224543    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs\") pod \"network-metrics-daemon-fdxms\" (UID: \"94979e3c-1ac8-4800-a60f-3d3552644bcc\") " pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:16.224727 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:16.224648    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:59:16.224727 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:16.224708    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret podName:c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:48.224693883 +0000 UTC m=+65.233623406 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret") pod "global-pull-secret-syncer-58crn" (UID: "c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:59:16.224727 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:16.224706    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:59:16.224831 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:16.224754    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs podName:94979e3c-1ac8-4800-a60f-3d3552644bcc nodeName:}" failed. No retries permitted until 2026-09-17 21:59:48.224741522 +0000 UTC m=+65.233671045 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs") pod "network-metrics-daemon-fdxms" (UID: "94979e3c-1ac8-4800-a60f-3d3552644bcc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:59:16.634623 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:16.634569    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:16.634623 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:16.634569    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:16.634798 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:16.634668    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-th2bd" podUID="da365699-cc35-49b9-a86a-58abbcc127f2"
Sep 17 21:59:16.634798 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:16.634740    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-58crn" podUID="c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673"
Sep 17 21:59:16.949207 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:16.949125    2567 generic.go:356] "Generic (PLEG): container finished" podID="d856ebe3-2d7c-4833-9576-504511abb585" containerID="d705d60ec166fc688b0e5fad2f473af85f461b41cefb631a522d21c8b2358820" exitCode=0
Sep 17 21:59:16.949207 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:16.949173    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9nmn" event={"ID":"d856ebe3-2d7c-4833-9576-504511abb585","Type":"ContainerDied","Data":"d705d60ec166fc688b0e5fad2f473af85f461b41cefb631a522d21c8b2358820"}
Sep 17 21:59:17.265561 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.265531    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-197.ec2.internal" event="NodeReady"
Sep 17 21:59:17.265697 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.265637    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 21:59:17.300280 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.300255    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79f5f54446-9cprq"]
Sep 17 21:59:17.312757 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.312736    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.315354 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.315330    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 21:59:17.315480 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.315376    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4d58n"]
Sep 17 21:59:17.316529 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.316494    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 21:59:17.316666 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.316647    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h2m4k\""
Sep 17 21:59:17.316850 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.316832    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 21:59:17.323369 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.323352    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 21:59:17.328853 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.328836    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79f5f54446-9cprq"]
Sep 17 21:59:17.328918 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.328863    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n7wmb"]
Sep 17 21:59:17.328978 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.328961    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4d58n"
Sep 17 21:59:17.331448 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.331304    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 21:59:17.331448 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.331386    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 21:59:17.331641 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.331499    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 21:59:17.331855 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.331840    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4j597\""
Sep 17 21:59:17.351180 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.351158    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4d58n"]
Sep 17 21:59:17.351260 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.351186    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n7wmb"]
Sep 17 21:59:17.351304 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.351266    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:17.353397 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.353376    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 21:59:17.353515 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.353472    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 21:59:17.353582 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.353515    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mg4m2\""
Sep 17 21:59:17.436749 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.436722    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39ecffbe-80f7-449f-9b29-dca2ffe07044-image-registry-private-configuration\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.436835 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.436753    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-certificates\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.436835 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.436770    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-bound-sa-token\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.436835 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.436784    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7-cert\") pod \"ingress-canary-4d58n\" (UID: \"e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7\") " pod="openshift-ingress-canary/ingress-canary-4d58n"
Sep 17 21:59:17.436835 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.436814    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.437069 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.436862    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2bs8\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-kube-api-access-k2bs8\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.437069 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.436929    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:17.437069 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.437016    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lztft\" (UniqueName: \"kubernetes.io/projected/e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7-kube-api-access-lztft\") pod \"ingress-canary-4d58n\" (UID: \"e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7\") " pod="openshift-ingress-canary/ingress-canary-4d58n"
Sep 17 21:59:17.437069 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.437050    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39ecffbe-80f7-449f-9b29-dca2ffe07044-installation-pull-secrets\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.437212 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.437073    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc28q\" (UniqueName: \"kubernetes.io/projected/5d9abd2c-c138-4356-963c-160dfdb23d5a-kube-api-access-fc28q\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:17.437212 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.437128    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39ecffbe-80f7-449f-9b29-dca2ffe07044-trusted-ca\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.437212 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.437155    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9abd2c-c138-4356-963c-160dfdb23d5a-config-volume\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:17.437212 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.437170    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9abd2c-c138-4356-963c-160dfdb23d5a-tmp-dir\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:17.437212 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.437189    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39ecffbe-80f7-449f-9b29-dca2ffe07044-ca-trust-extracted\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.538181 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538139    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lztft\" (UniqueName: \"kubernetes.io/projected/e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7-kube-api-access-lztft\") pod \"ingress-canary-4d58n\" (UID: \"e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7\") " pod="openshift-ingress-canary/ingress-canary-4d58n"
Sep 17 21:59:17.538181 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538165    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39ecffbe-80f7-449f-9b29-dca2ffe07044-installation-pull-secrets\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.538299 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538182    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc28q\" (UniqueName: \"kubernetes.io/projected/5d9abd2c-c138-4356-963c-160dfdb23d5a-kube-api-access-fc28q\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:17.538299 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538211    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39ecffbe-80f7-449f-9b29-dca2ffe07044-trusted-ca\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.538299 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538233    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9abd2c-c138-4356-963c-160dfdb23d5a-config-volume\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:17.538299 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538255    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9abd2c-c138-4356-963c-160dfdb23d5a-tmp-dir\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:17.538511 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538395    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39ecffbe-80f7-449f-9b29-dca2ffe07044-ca-trust-extracted\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.538511 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538445    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39ecffbe-80f7-449f-9b29-dca2ffe07044-image-registry-private-configuration\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.538511 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538494    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-certificates\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.538661 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538533    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-bound-sa-token\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.538661 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538542    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d9abd2c-c138-4356-963c-160dfdb23d5a-tmp-dir\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:17.538661 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538561    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7-cert\") pod \"ingress-canary-4d58n\" (UID: \"e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7\") " pod="openshift-ingress-canary/ingress-canary-4d58n"
Sep 17 21:59:17.538661 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538614    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.538661 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538641    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2bs8\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-kube-api-access-k2bs8\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.538878 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538678    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:17.538878 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:17.538792    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:17.538878 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.538825    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9abd2c-c138-4356-963c-160dfdb23d5a-config-volume\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:17.538878 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:17.538853    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls podName:5d9abd2c-c138-4356-963c-160dfdb23d5a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:18.038836821 +0000 UTC m=+35.047766344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls") pod "dns-default-n7wmb" (UID: "5d9abd2c-c138-4356-963c-160dfdb23d5a") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:17.539020 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:17.538909    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:59:17.539020 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:17.538920    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f5f54446-9cprq: secret "image-registry-tls" not found
Sep 17 21:59:17.539020 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:17.538953    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls podName:39ecffbe-80f7-449f-9b29-dca2ffe07044 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:18.038943302 +0000 UTC m=+35.047872823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls") pod "image-registry-79f5f54446-9cprq" (UID: "39ecffbe-80f7-449f-9b29-dca2ffe07044") : secret "image-registry-tls" not found
Sep 17 21:59:17.539121 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.539022    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-certificates\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.539240 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.539222    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39ecffbe-80f7-449f-9b29-dca2ffe07044-trusted-ca\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.542341 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.542245    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39ecffbe-80f7-449f-9b29-dca2ffe07044-image-registry-private-configuration\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.542414 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.542260    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7-cert\") pod \"ingress-canary-4d58n\" (UID: \"e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7\") " pod="openshift-ingress-canary/ingress-canary-4d58n"
Sep 17 21:59:17.542414 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.542281    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39ecffbe-80f7-449f-9b29-dca2ffe07044-installation-pull-secrets\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.545866 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.545837    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lztft\" (UniqueName: \"kubernetes.io/projected/e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7-kube-api-access-lztft\") pod \"ingress-canary-4d58n\" (UID: \"e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7\") " pod="openshift-ingress-canary/ingress-canary-4d58n"
Sep 17 21:59:17.545947 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.545891    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc28q\" (UniqueName: \"kubernetes.io/projected/5d9abd2c-c138-4356-963c-160dfdb23d5a-kube-api-access-fc28q\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:17.552792 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.552774    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39ecffbe-80f7-449f-9b29-dca2ffe07044-ca-trust-extracted\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.554408 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.554392    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-bound-sa-token\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.554530 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.554515    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2bs8\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-kube-api-access-k2bs8\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:17.634742 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.634722    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:17.637320 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.637301    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:59:17.637396 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.637307    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9ks7q\""
Sep 17 21:59:17.637575 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.637561    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4d58n"
Sep 17 21:59:17.954977 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.954890    2567 generic.go:356] "Generic (PLEG): container finished" podID="d856ebe3-2d7c-4833-9576-504511abb585" containerID="18e3e3aa88ec9a7e6eaf433d9ac4eae731a755658fb834be05c93dc64a6d1f1c" exitCode=0
Sep 17 21:59:17.954977 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.954944    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9nmn" event={"ID":"d856ebe3-2d7c-4833-9576-504511abb585","Type":"ContainerDied","Data":"18e3e3aa88ec9a7e6eaf433d9ac4eae731a755658fb834be05c93dc64a6d1f1c"}
Sep 17 21:59:17.975320 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:17.975295    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4d58n"]
Sep 17 21:59:17.978470 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:59:17.978447    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48a9d5b_0ccd_4f73_8f2b_6e467fcf70c7.slice/crio-4b7865350ce5162d1d51e5f4d536908dc5e41af82f45156e0d1176e06d450a14 WatchSource:0}: Error finding container 4b7865350ce5162d1d51e5f4d536908dc5e41af82f45156e0d1176e06d450a14: Status 404 returned error can't find the container with id 4b7865350ce5162d1d51e5f4d536908dc5e41af82f45156e0d1176e06d450a14
Sep 17 21:59:18.041713 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.041687    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:18.041808 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.041724    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:18.041847 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:18.041813    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:18.041894 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:18.041866    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls podName:5d9abd2c-c138-4356-963c-160dfdb23d5a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:19.041850144 +0000 UTC m=+36.050779663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls") pod "dns-default-n7wmb" (UID: "5d9abd2c-c138-4356-963c-160dfdb23d5a") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:18.041967 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:18.041946    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:59:18.042028 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:18.041972    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f5f54446-9cprq: secret "image-registry-tls" not found
Sep 17 21:59:18.042083 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:18.042061    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls podName:39ecffbe-80f7-449f-9b29-dca2ffe07044 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:19.042043714 +0000 UTC m=+36.050973240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls") pod "image-registry-79f5f54446-9cprq" (UID: "39ecffbe-80f7-449f-9b29-dca2ffe07044") : secret "image-registry-tls" not found
Sep 17 21:59:18.634724 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.634694    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:18.634915 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.634898    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:18.637338 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.637316    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 21:59:18.637451 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.637333    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q7xxp\""
Sep 17 21:59:18.637451 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.637336    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 21:59:18.637451 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.637322    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:59:18.844700 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.844671    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c9h8t"]
Sep 17 21:59:18.877452 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.877419    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c9h8t"]
Sep 17 21:59:18.877605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.877525    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:18.879855 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.879673    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 21:59:18.879855 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.879681    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 21:59:18.879855 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.879726    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 21:59:18.879855 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.879771    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 21:59:18.880323 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.880210    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g6j2k\""
Sep 17 21:59:18.950324 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.950254    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5jvr\" (UniqueName: \"kubernetes.io/projected/0374325e-9608-4e5b-9551-0f226154fa74-kube-api-access-r5jvr\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:18.950324 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.950303    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0374325e-9608-4e5b-9551-0f226154fa74-data-volume\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:18.950497 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.950390    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:18.950497 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.950421    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0374325e-9608-4e5b-9551-0f226154fa74-crio-socket\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:18.950497 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.950460    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:18.959149 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.959120    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4d58n" event={"ID":"e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7","Type":"ContainerStarted","Data":"4b7865350ce5162d1d51e5f4d536908dc5e41af82f45156e0d1176e06d450a14"}
Sep 17 21:59:18.963729 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.963701    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9nmn" event={"ID":"d856ebe3-2d7c-4833-9576-504511abb585","Type":"ContainerStarted","Data":"9b57ca965b4a529e49c0bf05ab1db8ad69b0928050605716fde74d306d52804b"}
Sep 17 21:59:18.987710 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:18.987599    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q9nmn" podStartSLOduration=7.215330866 podStartE2EDuration="35.987583043s" totalImagesPullingTime="28.772252177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:18.986172646 +0000 UTC m=+35.995102188" watchObservedRunningTime="2026-09-17 21:59:18.987583043 +0000 UTC m=+35.996512586"
Sep 17 21:59:19.051515 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.051481    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:19.051515 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.051520    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0374325e-9608-4e5b-9551-0f226154fa74-crio-socket\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:19.051732 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.051600    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:19.051732 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:19.051650    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:19.051732 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.051671    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jvr\" (UniqueName: \"kubernetes.io/projected/0374325e-9608-4e5b-9551-0f226154fa74-kube-api-access-r5jvr\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:19.051732 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.051707    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0374325e-9608-4e5b-9551-0f226154fa74-data-volume\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:19.051732 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:19.051727    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm podName:0374325e-9608-4e5b-9551-0f226154fa74 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:19.551705862 +0000 UTC m=+36.560635398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c9h8t" (UID: "0374325e-9608-4e5b-9551-0f226154fa74") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:19.051985 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.051768    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:19.051985 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.051792    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0374325e-9608-4e5b-9551-0f226154fa74-crio-socket\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:19.051985 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:19.051815    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:19.051985 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.051801    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:19.051985 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:19.051867    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls podName:0374325e-9608-4e5b-9551-0f226154fa74 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:19.551848569 +0000 UTC m=+36.560778104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9h8t" (UID: "0374325e-9608-4e5b-9551-0f226154fa74") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:19.051985 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:19.051886    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:19.051985 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:19.051920    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls podName:5d9abd2c-c138-4356-963c-160dfdb23d5a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:21.051908782 +0000 UTC m=+38.060838320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls") pod "dns-default-n7wmb" (UID: "5d9abd2c-c138-4356-963c-160dfdb23d5a") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:19.051985 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.051968    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0374325e-9608-4e5b-9551-0f226154fa74-data-volume\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:19.052323 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:19.052103    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:59:19.052323 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:19.052114    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f5f54446-9cprq: secret "image-registry-tls" not found
Sep 17 21:59:19.052323 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:19.052161    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls podName:39ecffbe-80f7-449f-9b29-dca2ffe07044 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:21.052146687 +0000 UTC m=+38.061076213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls") pod "image-registry-79f5f54446-9cprq" (UID: "39ecffbe-80f7-449f-9b29-dca2ffe07044") : secret "image-registry-tls" not found
Sep 17 21:59:19.060799 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.060772    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5jvr\" (UniqueName: \"kubernetes.io/projected/0374325e-9608-4e5b-9551-0f226154fa74-kube-api-access-r5jvr\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:19.555412 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.555376    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:19.555605 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.555489    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:19.555605 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:19.555542    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:19.555721 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:19.555607    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:19.555721 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:19.555613    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls podName:0374325e-9608-4e5b-9551-0f226154fa74 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:20.555598019 +0000 UTC m=+37.564527542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9h8t" (UID: "0374325e-9608-4e5b-9551-0f226154fa74") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:19.555721 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:19.555661    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm podName:0374325e-9608-4e5b-9551-0f226154fa74 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:20.555647326 +0000 UTC m=+37.564576849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c9h8t" (UID: "0374325e-9608-4e5b-9551-0f226154fa74") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:19.841622 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.841548    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-wcht2"]
Sep 17 21:59:19.860362 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.860327    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-wcht2"]
Sep 17 21:59:19.860490 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.860458    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wcht2"
Sep 17 21:59:19.863259 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.863232    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 21:59:19.864238 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.864218    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-p6xnn\""
Sep 17 21:59:19.864238 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.864232    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 21:59:19.959429 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:19.959400    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvgjg\" (UniqueName: \"kubernetes.io/projected/9805e819-611c-4b5d-af80-a380d9004387-kube-api-access-fvgjg\") pod \"migrator-76794cbd6c-wcht2\" (UID: \"9805e819-611c-4b5d-af80-a380d9004387\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wcht2"
Sep 17 21:59:20.060831 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:20.060794    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvgjg\" (UniqueName: \"kubernetes.io/projected/9805e819-611c-4b5d-af80-a380d9004387-kube-api-access-fvgjg\") pod \"migrator-76794cbd6c-wcht2\" (UID: \"9805e819-611c-4b5d-af80-a380d9004387\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wcht2"
Sep 17 21:59:20.069638 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:20.069614    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvgjg\" (UniqueName: \"kubernetes.io/projected/9805e819-611c-4b5d-af80-a380d9004387-kube-api-access-fvgjg\") pod \"migrator-76794cbd6c-wcht2\" (UID: \"9805e819-611c-4b5d-af80-a380d9004387\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wcht2"
Sep 17 21:59:20.170528 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:20.170469    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wcht2"
Sep 17 21:59:20.390312 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:20.390293    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n4zhm_149b4a0e-4466-46a6-9e51-d2450702c4d0/dns-node-resolver/0.log"
Sep 17 21:59:20.449284 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:20.449261    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-wcht2"]
Sep 17 21:59:20.452126 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:59:20.452095    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9805e819_611c_4b5d_af80_a380d9004387.slice/crio-4d56e293e9b5484c673f9e428e0b786eeb1a2c2c283d489565672272fa80eb5a WatchSource:0}: Error finding container 4d56e293e9b5484c673f9e428e0b786eeb1a2c2c283d489565672272fa80eb5a: Status 404 returned error can't find the container with id 4d56e293e9b5484c673f9e428e0b786eeb1a2c2c283d489565672272fa80eb5a
Sep 17 21:59:20.563479 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:20.563451    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:20.563669 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:20.563498    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:20.563669 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:20.563596    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:20.563669 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:20.563613    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:20.563669 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:20.563670    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls podName:0374325e-9608-4e5b-9551-0f226154fa74 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:22.563652952 +0000 UTC m=+39.572582487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9h8t" (UID: "0374325e-9608-4e5b-9551-0f226154fa74") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:20.563818 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:20.563689    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm podName:0374325e-9608-4e5b-9551-0f226154fa74 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:22.563680876 +0000 UTC m=+39.572610409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c9h8t" (UID: "0374325e-9608-4e5b-9551-0f226154fa74") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:20.969602 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:20.969576    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4d58n" event={"ID":"e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7","Type":"ContainerStarted","Data":"c4a944629eaaa8c2a46ec80cd9f76a2e6a962831511d6807f801c6d83749e7db"}
Sep 17 21:59:20.970703 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:20.970684    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wcht2" event={"ID":"9805e819-611c-4b5d-af80-a380d9004387","Type":"ContainerStarted","Data":"4d56e293e9b5484c673f9e428e0b786eeb1a2c2c283d489565672272fa80eb5a"}
Sep 17 21:59:20.988434 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:20.988394    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4d58n" podStartSLOduration=1.589459674 podStartE2EDuration="3.988380883s" totalImagesPullingTime="2.398921209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:20.987728944 +0000 UTC m=+37.996658490" watchObservedRunningTime="2026-09-17 21:59:20.988380883 +0000 UTC m=+37.997310424"
Sep 17 21:59:21.067628 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.067592    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:21.067780 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.067636    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:21.067780 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:21.067737    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:21.067954 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:21.067780    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:59:21.067954 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:21.067799    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f5f54446-9cprq: secret "image-registry-tls" not found
Sep 17 21:59:21.067954 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:21.067791    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls podName:5d9abd2c-c138-4356-963c-160dfdb23d5a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:25.067774631 +0000 UTC m=+42.076704166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls") pod "dns-default-n7wmb" (UID: "5d9abd2c-c138-4356-963c-160dfdb23d5a") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:21.067954 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:21.067856    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls podName:39ecffbe-80f7-449f-9b29-dca2ffe07044 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:25.067839812 +0000 UTC m=+42.076769331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls") pod "image-registry-79f5f54446-9cprq" (UID: "39ecffbe-80f7-449f-9b29-dca2ffe07044") : secret "image-registry-tls" not found
Sep 17 21:59:21.278617 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.278586    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-wq799"]
Sep 17 21:59:21.298740 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.298718    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-wq799"]
Sep 17 21:59:21.298881 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.298813    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-wq799"
Sep 17 21:59:21.301296 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.301272    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 21:59:21.301447 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.301431    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 21:59:21.301794 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.301776    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 21:59:21.301794 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.301788    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8q4ww\""
Sep 17 21:59:21.301959 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.301863    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 21:59:21.371099 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.371069    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rzk\" (UniqueName: \"kubernetes.io/projected/5e620ba8-4e38-4d81-a8b2-adc3d8414b5f-kube-api-access-68rzk\") pod \"service-ca-55f6fc7b86-wq799\" (UID: \"5e620ba8-4e38-4d81-a8b2-adc3d8414b5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wq799"
Sep 17 21:59:21.371205 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.371121    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5e620ba8-4e38-4d81-a8b2-adc3d8414b5f-signing-key\") pod \"service-ca-55f6fc7b86-wq799\" (UID: \"5e620ba8-4e38-4d81-a8b2-adc3d8414b5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wq799"
Sep 17 21:59:21.371205 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.371179    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5e620ba8-4e38-4d81-a8b2-adc3d8414b5f-signing-cabundle\") pod \"service-ca-55f6fc7b86-wq799\" (UID: \"5e620ba8-4e38-4d81-a8b2-adc3d8414b5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wq799"
Sep 17 21:59:21.472506 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.472480    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rzk\" (UniqueName: \"kubernetes.io/projected/5e620ba8-4e38-4d81-a8b2-adc3d8414b5f-kube-api-access-68rzk\") pod \"service-ca-55f6fc7b86-wq799\" (UID: \"5e620ba8-4e38-4d81-a8b2-adc3d8414b5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wq799"
Sep 17 21:59:21.472658 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.472546    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5e620ba8-4e38-4d81-a8b2-adc3d8414b5f-signing-key\") pod \"service-ca-55f6fc7b86-wq799\" (UID: \"5e620ba8-4e38-4d81-a8b2-adc3d8414b5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wq799"
Sep 17 21:59:21.472658 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.472580    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5e620ba8-4e38-4d81-a8b2-adc3d8414b5f-signing-cabundle\") pod \"service-ca-55f6fc7b86-wq799\" (UID: \"5e620ba8-4e38-4d81-a8b2-adc3d8414b5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wq799"
Sep 17 21:59:21.473249 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.473231    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5e620ba8-4e38-4d81-a8b2-adc3d8414b5f-signing-cabundle\") pod \"service-ca-55f6fc7b86-wq799\" (UID: \"5e620ba8-4e38-4d81-a8b2-adc3d8414b5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wq799"
Sep 17 21:59:21.475863 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.475835    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5e620ba8-4e38-4d81-a8b2-adc3d8414b5f-signing-key\") pod \"service-ca-55f6fc7b86-wq799\" (UID: \"5e620ba8-4e38-4d81-a8b2-adc3d8414b5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wq799"
Sep 17 21:59:21.481319 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.481302    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rzk\" (UniqueName: \"kubernetes.io/projected/5e620ba8-4e38-4d81-a8b2-adc3d8414b5f-kube-api-access-68rzk\") pod \"service-ca-55f6fc7b86-wq799\" (UID: \"5e620ba8-4e38-4d81-a8b2-adc3d8414b5f\") " pod="openshift-service-ca/service-ca-55f6fc7b86-wq799"
Sep 17 21:59:21.590573 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.590506    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xc5tg_5dd82d87-0321-4c10-a183-6d6122eda0cc/node-ca/0.log"
Sep 17 21:59:21.610752 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.610728    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-wq799"
Sep 17 21:59:21.980111 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.979930    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-wq799"]
Sep 17 21:59:21.990769 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:21.990738    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4d58n_e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7/serve-healthcheck-canary/0.log"
Sep 17 21:59:22.015402 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:59:22.015378    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e620ba8_4e38_4d81_a8b2_adc3d8414b5f.slice/crio-cd7667c28d4756ec761b00b0cbe9ca93f39b274881e66053163b33d0181a3c03 WatchSource:0}: Error finding container cd7667c28d4756ec761b00b0cbe9ca93f39b274881e66053163b33d0181a3c03: Status 404 returned error can't find the container with id cd7667c28d4756ec761b00b0cbe9ca93f39b274881e66053163b33d0181a3c03
Sep 17 21:59:22.583374 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:22.583305    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:22.583374 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:22.583360    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:22.583556 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:22.583425    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:22.583556 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:22.583494    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm podName:0374325e-9608-4e5b-9551-0f226154fa74 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:26.58347896 +0000 UTC m=+43.592408480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c9h8t" (UID: "0374325e-9608-4e5b-9551-0f226154fa74") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:22.583556 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:22.583516    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:22.583698 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:22.583584    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls podName:0374325e-9608-4e5b-9551-0f226154fa74 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:26.583567582 +0000 UTC m=+43.592497123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9h8t" (UID: "0374325e-9608-4e5b-9551-0f226154fa74") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:22.978178 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:22.978093    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wcht2" event={"ID":"9805e819-611c-4b5d-af80-a380d9004387","Type":"ContainerStarted","Data":"4d300977043cfec06610ee52f8d605d70103019ae31c2dbd69ca7c34c8ccb412"}
Sep 17 21:59:22.978321 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:22.978191    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wcht2" event={"ID":"9805e819-611c-4b5d-af80-a380d9004387","Type":"ContainerStarted","Data":"660fce2d21d6a68e8677043831ee5419a5d663c14bbb401fdf5378f0005f8208"}
Sep 17 21:59:22.979336 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:22.979312    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-wq799" event={"ID":"5e620ba8-4e38-4d81-a8b2-adc3d8414b5f","Type":"ContainerStarted","Data":"cd7667c28d4756ec761b00b0cbe9ca93f39b274881e66053163b33d0181a3c03"}
Sep 17 21:59:23.004344 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:23.004296    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-wcht2" podStartSLOduration=2.407207746 podStartE2EDuration="4.004279106s" totalImagesPullingTime="1.59707136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:23.003911203 +0000 UTC m=+40.012840746" watchObservedRunningTime="2026-09-17 21:59:23.004279106 +0000 UTC m=+40.013208647"
Sep 17 21:59:24.985907 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:24.985878    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-wq799" event={"ID":"5e620ba8-4e38-4d81-a8b2-adc3d8414b5f","Type":"ContainerStarted","Data":"393d73d716b7cc006c7286d03b262bbeda2353fa320a844a43fd6c1a69bff63f"}
Sep 17 21:59:25.002234 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:25.002190    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-wq799" podStartSLOduration=1.510864227 podStartE2EDuration="4.002175278s" totalImagesPullingTime="2.491311051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:25.001526689 +0000 UTC m=+42.010456231" watchObservedRunningTime="2026-09-17 21:59:25.002175278 +0000 UTC m=+42.011104821"
Sep 17 21:59:25.104073 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:25.104045    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:25.104182 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:25.104077    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:25.104244 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:25.104190    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:25.104292 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:25.104248    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls podName:5d9abd2c-c138-4356-963c-160dfdb23d5a nodeName:}" failed. No retries permitted until 2026-09-17 21:59:33.104233864 +0000 UTC m=+50.113163396 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls") pod "dns-default-n7wmb" (UID: "5d9abd2c-c138-4356-963c-160dfdb23d5a") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:25.104292 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:25.104194    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 21:59:25.104396 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:25.104292    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f5f54446-9cprq: secret "image-registry-tls" not found
Sep 17 21:59:25.104396 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:25.104331    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls podName:39ecffbe-80f7-449f-9b29-dca2ffe07044 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:33.104317348 +0000 UTC m=+50.113246871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls") pod "image-registry-79f5f54446-9cprq" (UID: "39ecffbe-80f7-449f-9b29-dca2ffe07044") : secret "image-registry-tls" not found
Sep 17 21:59:26.615503 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:26.615468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:26.615847 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:26.615528    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:26.615847 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:26.615597    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:26.615847 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:26.615657    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm podName:0374325e-9608-4e5b-9551-0f226154fa74 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:34.61564229 +0000 UTC m=+51.624571829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c9h8t" (UID: "0374325e-9608-4e5b-9551-0f226154fa74") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:26.615847 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:26.615695    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:26.615847 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:26.615749    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls podName:0374325e-9608-4e5b-9551-0f226154fa74 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:34.615736779 +0000 UTC m=+51.624666302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9h8t" (UID: "0374325e-9608-4e5b-9551-0f226154fa74") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:33.158284 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:33.158242    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:33.158284 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:33.158285    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:33.161157 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:33.161121    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5d9abd2c-c138-4356-963c-160dfdb23d5a-metrics-tls\") pod \"dns-default-n7wmb\" (UID: \"5d9abd2c-c138-4356-963c-160dfdb23d5a\") " pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:33.161272 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:33.161242    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls\") pod \"image-registry-79f5f54446-9cprq\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") " pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:33.222647 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:33.222611    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:33.258839 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:33.258821    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:33.532660 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:33.532629    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79f5f54446-9cprq"]
Sep 17 21:59:33.535131 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:59:33.535105    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ecffbe_80f7_449f_9b29_dca2ffe07044.slice/crio-36f99ed652eee4f3c22a1330d1775193fe6998c25b43b2d379cdd8330519c0a0 WatchSource:0}: Error finding container 36f99ed652eee4f3c22a1330d1775193fe6998c25b43b2d379cdd8330519c0a0: Status 404 returned error can't find the container with id 36f99ed652eee4f3c22a1330d1775193fe6998c25b43b2d379cdd8330519c0a0
Sep 17 21:59:33.550315 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:33.550292    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n7wmb"]
Sep 17 21:59:33.553184 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:59:33.553138    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9abd2c_c138_4356_963c_160dfdb23d5a.slice/crio-fc2a19100150c3079cb978cdfd0c20285f3dfed85b243beae05e2f19997c4e08 WatchSource:0}: Error finding container fc2a19100150c3079cb978cdfd0c20285f3dfed85b243beae05e2f19997c4e08: Status 404 returned error can't find the container with id fc2a19100150c3079cb978cdfd0c20285f3dfed85b243beae05e2f19997c4e08
Sep 17 21:59:34.014853 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:34.012572    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n7wmb" event={"ID":"5d9abd2c-c138-4356-963c-160dfdb23d5a","Type":"ContainerStarted","Data":"fc2a19100150c3079cb978cdfd0c20285f3dfed85b243beae05e2f19997c4e08"}
Sep 17 21:59:34.014853 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:34.014674    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f5f54446-9cprq" event={"ID":"39ecffbe-80f7-449f-9b29-dca2ffe07044","Type":"ContainerStarted","Data":"0fb1eea9fbc4144e2af05b496339161e6205f054378d6d9ae36ee840a005613d"}
Sep 17 21:59:34.014853 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:34.014716    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f5f54446-9cprq" event={"ID":"39ecffbe-80f7-449f-9b29-dca2ffe07044","Type":"ContainerStarted","Data":"36f99ed652eee4f3c22a1330d1775193fe6998c25b43b2d379cdd8330519c0a0"}
Sep 17 21:59:34.014853 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:34.014766    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 21:59:34.038803 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:34.038754    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79f5f54446-9cprq" podStartSLOduration=23.038734265 podStartE2EDuration="23.038734265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:34.037233766 +0000 UTC m=+51.046163309" watchObservedRunningTime="2026-09-17 21:59:34.038734265 +0000 UTC m=+51.047663807"
Sep 17 21:59:34.671925 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:34.671778    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:34.671925 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:34.671879    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:34.671925 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:34.671918    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:34.672463 ip-10-0-142-197 kubenswrapper[2567]: E0917 21:59:34.672015    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls podName:0374325e-9608-4e5b-9551-0f226154fa74 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:50.671977502 +0000 UTC m=+67.680907026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9h8t" (UID: "0374325e-9608-4e5b-9551-0f226154fa74") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:34.672463 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:34.672410    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0374325e-9608-4e5b-9551-0f226154fa74-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:36.022651 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:36.022617    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n7wmb" event={"ID":"5d9abd2c-c138-4356-963c-160dfdb23d5a","Type":"ContainerStarted","Data":"cdb32ecbe9ba09865b853b3fbde760d4b1573a4d66a7b3de9b9d6e598c7c6c79"}
Sep 17 21:59:36.023053 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:36.022673    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n7wmb" event={"ID":"5d9abd2c-c138-4356-963c-160dfdb23d5a","Type":"ContainerStarted","Data":"92d95374d3a3d7e691de61b7a37228a0b8ef5cbc66f5fca529cd7ba1874bac27"}
Sep 17 21:59:36.023053 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:36.022762    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:36.040487 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:36.040440    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n7wmb" podStartSLOduration=17.470520951 podStartE2EDuration="19.040424093s" totalImagesPullingTime="1.569903142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:36.039423633 +0000 UTC m=+53.048353174" watchObservedRunningTime="2026-09-17 21:59:36.040424093 +0000 UTC m=+53.049353634"
Sep 17 21:59:40.136078 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.136047    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"]
Sep 17 21:59:40.157304 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.157280    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"]
Sep 17 21:59:40.157451 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.157414    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"
Sep 17 21:59:40.160290 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.160267    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 21:59:40.160398 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.160324    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 21:59:40.160398 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.160361    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 21:59:40.160924 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.160904    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 21:59:40.170873 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.170855    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"]
Sep 17 21:59:40.170963 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.170886    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79f5f54446-9cprq"]
Sep 17 21:59:40.170963 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.170904    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"]
Sep 17 21:59:40.171057 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.170968    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.173637 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.173619    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 21:59:40.173962 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.173944    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 21:59:40.174061 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.173947    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 21:59:40.174061 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.173976    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 21:59:40.246589 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.246561    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fdccdf588-rpvrh"]
Sep 17 21:59:40.266375 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.266355    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fdccdf588-rpvrh"]
Sep 17 21:59:40.266470 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.266428    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.314398 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.314370    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.314513 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.314403    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-ca\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.314513 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.314429    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt9ck\" (UniqueName: \"kubernetes.io/projected/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-kube-api-access-nt9ck\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.314593 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.314530    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4ldg\" (UniqueName: \"kubernetes.io/projected/b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6-kube-api-access-f4ldg\") pod \"klusterlet-addon-workmgr-7689959575-mr8m4\" (UID: \"b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"
Sep 17 21:59:40.314593 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.314555    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.314593 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.314573    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-hub\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.314702 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.314621    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.314702 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.314648    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6-klusterlet-config\") pod \"klusterlet-addon-workmgr-7689959575-mr8m4\" (UID: \"b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"
Sep 17 21:59:40.314702 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.314678    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6-tmp\") pod \"klusterlet-addon-workmgr-7689959575-mr8m4\" (UID: \"b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"
Sep 17 21:59:40.415131 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415072    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.415131 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415103    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-bound-sa-token\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.415298 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415134    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-ca-trust-extracted\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.415298 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415158    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6-klusterlet-config\") pod \"klusterlet-addon-workmgr-7689959575-mr8m4\" (UID: \"b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"
Sep 17 21:59:40.415298 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415180    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6-tmp\") pod \"klusterlet-addon-workmgr-7689959575-mr8m4\" (UID: \"b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"
Sep 17 21:59:40.415298 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415217    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-image-registry-private-configuration\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.415298 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415261    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.415298 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415285    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-ca\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.415677 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415310    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt9ck\" (UniqueName: \"kubernetes.io/projected/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-kube-api-access-nt9ck\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.415677 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415354    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-installation-pull-secrets\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.415677 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415430    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-registry-certificates\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.415677 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415458    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-registry-tls\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.415677 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415507    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4ldg\" (UniqueName: \"kubernetes.io/projected/b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6-kube-api-access-f4ldg\") pod \"klusterlet-addon-workmgr-7689959575-mr8m4\" (UID: \"b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"
Sep 17 21:59:40.415677 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415531    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-trusted-ca\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.415677 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415561    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.415677 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415589    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-hub\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.415677 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415614    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4bs2\" (UniqueName: \"kubernetes.io/projected/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-kube-api-access-g4bs2\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.416071 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.415975    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.417583 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.417562    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.424185 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.424167    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt9ck\" (UniqueName: \"kubernetes.io/projected/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-kube-api-access-nt9ck\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.425895 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.425873    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6-tmp\") pod \"klusterlet-addon-workmgr-7689959575-mr8m4\" (UID: \"b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"
Sep 17 21:59:40.426477 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.426440    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6-klusterlet-config\") pod \"klusterlet-addon-workmgr-7689959575-mr8m4\" (UID: \"b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"
Sep 17 21:59:40.427775 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.427748    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.427854 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.427772    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-ca\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.427930 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.427912    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/86e5fd2b-3100-4d2e-a7bd-3622a7834a4d-hub\") pod \"cluster-proxy-proxy-agent-57dc97d7d8-hv8qk\" (UID: \"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.428366 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.428348    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4ldg\" (UniqueName: \"kubernetes.io/projected/b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6-kube-api-access-f4ldg\") pod \"klusterlet-addon-workmgr-7689959575-mr8m4\" (UID: \"b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"
Sep 17 21:59:40.468072 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.468051    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"
Sep 17 21:59:40.491982 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.491963    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"
Sep 17 21:59:40.516245 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.516219    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-registry-certificates\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.516356 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.516250    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-registry-tls\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.516356 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.516272    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-trusted-ca\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.516356 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.516295    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4bs2\" (UniqueName: \"kubernetes.io/projected/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-kube-api-access-g4bs2\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.516503 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.516416    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-bound-sa-token\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.516503 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.516433    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-ca-trust-extracted\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.516503 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.516472    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-image-registry-private-configuration\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.516649 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.516526    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-installation-pull-secrets\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.517114 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.517068    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-ca-trust-extracted\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.517596 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.517573    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-trusted-ca\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.517769 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.517742    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-registry-certificates\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.518900 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.518879    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-registry-tls\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.519078 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.519058    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-installation-pull-secrets\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.519231 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.519216    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-image-registry-private-configuration\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.524547 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.524516    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4bs2\" (UniqueName: \"kubernetes.io/projected/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-kube-api-access-g4bs2\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.524933 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.524914    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a2fb062-67ac-4fc4-85dc-a2ac4accf796-bound-sa-token\") pod \"image-registry-6fdccdf588-rpvrh\" (UID: \"7a2fb062-67ac-4fc4-85dc-a2ac4accf796\") " pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.574447 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.574422    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:40.760793 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.760763    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"]
Sep 17 21:59:40.763694 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:59:40.763668    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b2499e_3b6c_4be2_8a1f_1e41d57af2f6.slice/crio-0dfe5972f7b78a9ab18d6cd9a59d3223e5734419d739f4c10778a4e6ac7aa69b WatchSource:0}: Error finding container 0dfe5972f7b78a9ab18d6cd9a59d3223e5734419d739f4c10778a4e6ac7aa69b: Status 404 returned error can't find the container with id 0dfe5972f7b78a9ab18d6cd9a59d3223e5734419d739f4c10778a4e6ac7aa69b
Sep 17 21:59:40.780083 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.779978    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk"]
Sep 17 21:59:40.782309 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:59:40.782283    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86e5fd2b_3100_4d2e_a7bd_3622a7834a4d.slice/crio-31e01dad66832436aecfc9df08e2081018e305821c3280ba32dc48e387f0c34f WatchSource:0}: Error finding container 31e01dad66832436aecfc9df08e2081018e305821c3280ba32dc48e387f0c34f: Status 404 returned error can't find the container with id 31e01dad66832436aecfc9df08e2081018e305821c3280ba32dc48e387f0c34f
Sep 17 21:59:40.869764 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:40.869734    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fdccdf588-rpvrh"]
Sep 17 21:59:40.873100 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:59:40.873078    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2fb062_67ac_4fc4_85dc_a2ac4accf796.slice/crio-fd45d1314b05c829cf713a793696561dac324603c6706350714365ea5d76dacc WatchSource:0}: Error finding container fd45d1314b05c829cf713a793696561dac324603c6706350714365ea5d76dacc: Status 404 returned error can't find the container with id fd45d1314b05c829cf713a793696561dac324603c6706350714365ea5d76dacc
Sep 17 21:59:41.040328 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:41.040265    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk" event={"ID":"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d","Type":"ContainerStarted","Data":"31e01dad66832436aecfc9df08e2081018e305821c3280ba32dc48e387f0c34f"}
Sep 17 21:59:41.041838 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:41.041815    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh" event={"ID":"7a2fb062-67ac-4fc4-85dc-a2ac4accf796","Type":"ContainerStarted","Data":"44c03d65882b97913a4792b4a3bbd923b2b5e6440e33f0ca805678630020063a"}
Sep 17 21:59:41.041959 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:41.041858    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh" event={"ID":"7a2fb062-67ac-4fc4-85dc-a2ac4accf796","Type":"ContainerStarted","Data":"fd45d1314b05c829cf713a793696561dac324603c6706350714365ea5d76dacc"}
Sep 17 21:59:41.041959 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:41.041915    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 21:59:41.043040 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:41.043020    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4" event={"ID":"b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6","Type":"ContainerStarted","Data":"0dfe5972f7b78a9ab18d6cd9a59d3223e5734419d739f4c10778a4e6ac7aa69b"}
Sep 17 21:59:41.064593 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:41.064540    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh" podStartSLOduration=1.064522028 podStartE2EDuration="1.064522028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:41.063820212 +0000 UTC m=+58.072749768" watchObservedRunningTime="2026-09-17 21:59:41.064522028 +0000 UTC m=+58.073451571"
Sep 17 21:59:43.950854 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:43.950809    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2kp5t"
Sep 17 21:59:45.063336 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:45.063303    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4" event={"ID":"b6b2499e-3b6c-4be2-8a1f-1e41d57af2f6","Type":"ContainerStarted","Data":"bc82c564821223a1a4e4e851e574cb04cde47dbaf4084374c830ca86e9f52ae4"}
Sep 17 21:59:45.063786 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:45.063505    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"
Sep 17 21:59:45.065094 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:45.065065    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4"
Sep 17 21:59:45.081597 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:45.081558    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7689959575-mr8m4" podStartSLOduration=1.505620958 podStartE2EDuration="5.081545131s" totalImagesPullingTime="3.575924173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:45.080301931 +0000 UTC m=+62.089231485" watchObservedRunningTime="2026-09-17 21:59:45.081545131 +0000 UTC m=+62.090474672"
Sep 17 21:59:46.029202 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:46.029173    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n7wmb"
Sep 17 21:59:48.171501 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.171460    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxxx\" (UniqueName: \"kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx\") pod \"network-check-target-th2bd\" (UID: \"da365699-cc35-49b9-a86a-58abbcc127f2\") " pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:48.174262 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.174234    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 21:59:48.184072 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.184056    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 21:59:48.194444 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.194428    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwxxx\" (UniqueName: \"kubernetes.io/projected/da365699-cc35-49b9-a86a-58abbcc127f2-kube-api-access-bwxxx\") pod \"network-check-target-th2bd\" (UID: \"da365699-cc35-49b9-a86a-58abbcc127f2\") " pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:48.272789 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.272760    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs\") pod \"network-metrics-daemon-fdxms\" (UID: \"94979e3c-1ac8-4800-a60f-3d3552644bcc\") " pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:48.272903 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.272806    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:48.275308 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.275287    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:59:48.275407 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.275343    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:59:48.285363 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.285343    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94979e3c-1ac8-4800-a60f-3d3552644bcc-metrics-certs\") pod \"network-metrics-daemon-fdxms\" (UID: \"94979e3c-1ac8-4800-a60f-3d3552644bcc\") " pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:48.285442 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.285361    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673-original-pull-secret\") pod \"global-pull-secret-syncer-58crn\" (UID: \"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673\") " pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:48.348264 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.348242    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q7xxp\""
Sep 17 21:59:48.350069 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.350055    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-58crn"
Sep 17 21:59:48.355964 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.355950    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:48.545124 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.545101    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9ks7q\""
Sep 17 21:59:48.553051 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.553033    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fdxms"
Sep 17 21:59:48.625459 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.625434    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-58crn"]
Sep 17 21:59:48.627843 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.627822    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-th2bd"]
Sep 17 21:59:48.628677 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:59:48.628656    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02d81d0_2aeb_4cc8_a3e4_cc374c6bd673.slice/crio-976961fd451fe7933fcee6e7a34e9e869a2fda2e38e93ab037648ab06f5c448f WatchSource:0}: Error finding container 976961fd451fe7933fcee6e7a34e9e869a2fda2e38e93ab037648ab06f5c448f: Status 404 returned error can't find the container with id 976961fd451fe7933fcee6e7a34e9e869a2fda2e38e93ab037648ab06f5c448f
Sep 17 21:59:48.630409 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:59:48.630321    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda365699_cc35_49b9_a86a_58abbcc127f2.slice/crio-6921288f2b92f277cc4334d98f3cdad5feb64e217d3ae21e4f715318116e8e7f WatchSource:0}: Error finding container 6921288f2b92f277cc4334d98f3cdad5feb64e217d3ae21e4f715318116e8e7f: Status 404 returned error can't find the container with id 6921288f2b92f277cc4334d98f3cdad5feb64e217d3ae21e4f715318116e8e7f
Sep 17 21:59:48.816489 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:48.816353    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fdxms"]
Sep 17 21:59:48.818768 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:59:48.818746    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94979e3c_1ac8_4800_a60f_3d3552644bcc.slice/crio-12b1f23bcfe5f3cb13ac9fc31ce9fbb2f53cfb6f23aceabd9a5ca011f3070856 WatchSource:0}: Error finding container 12b1f23bcfe5f3cb13ac9fc31ce9fbb2f53cfb6f23aceabd9a5ca011f3070856: Status 404 returned error can't find the container with id 12b1f23bcfe5f3cb13ac9fc31ce9fbb2f53cfb6f23aceabd9a5ca011f3070856
Sep 17 21:59:49.078262 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:49.078176    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-th2bd" event={"ID":"da365699-cc35-49b9-a86a-58abbcc127f2","Type":"ContainerStarted","Data":"6921288f2b92f277cc4334d98f3cdad5feb64e217d3ae21e4f715318116e8e7f"}
Sep 17 21:59:49.080258 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:49.080225    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fdxms" event={"ID":"94979e3c-1ac8-4800-a60f-3d3552644bcc","Type":"ContainerStarted","Data":"12b1f23bcfe5f3cb13ac9fc31ce9fbb2f53cfb6f23aceabd9a5ca011f3070856"}
Sep 17 21:59:49.082164 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:49.082132    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-58crn" event={"ID":"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673","Type":"ContainerStarted","Data":"976961fd451fe7933fcee6e7a34e9e869a2fda2e38e93ab037648ab06f5c448f"}
Sep 17 21:59:50.088469 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:50.088354    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fdxms" event={"ID":"94979e3c-1ac8-4800-a60f-3d3552644bcc","Type":"ContainerStarted","Data":"ec562d9c9680a51271bd1cd004b5738cd38aba2fa02ff4ae7ce3e21ea235d2b2"}
Sep 17 21:59:50.177972 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:50.177923    2567 patch_prober.go:28] interesting pod/image-registry-79f5f54446-9cprq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 21:59:50.178178 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:50.177989    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-79f5f54446-9cprq" podUID="39ecffbe-80f7-449f-9b29-dca2ffe07044" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:59:50.692511 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:50.692469    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:50.695220 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:50.695194    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0374325e-9608-4e5b-9551-0f226154fa74-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9h8t\" (UID: \"0374325e-9608-4e5b-9551-0f226154fa74\") " pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:50.991478 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:50.991314    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g6j2k\""
Sep 17 21:59:50.997633 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:50.997605    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c9h8t"
Sep 17 21:59:51.094192 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:51.094154    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fdxms" event={"ID":"94979e3c-1ac8-4800-a60f-3d3552644bcc","Type":"ContainerStarted","Data":"d59139109d4c27a90447a0b0f8e24b70aab6cb7e9f78d6653e9f734d0afae214"}
Sep 17 21:59:51.110535 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:51.110482    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fdxms" podStartSLOduration=66.994569721 podStartE2EDuration="1m8.110462535s" totalImagesPullingTime="1.115892814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:51.109324372 +0000 UTC m=+68.118253926" watchObservedRunningTime="2026-09-17 21:59:51.110462535 +0000 UTC m=+68.119392075"
Sep 17 21:59:53.104285 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:53.104254    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-th2bd" event={"ID":"da365699-cc35-49b9-a86a-58abbcc127f2","Type":"ContainerStarted","Data":"7e26f58ff044168cbc2ce0a32e628143452f399a9def85d435bdd683086e2cab"}
Sep 17 21:59:53.104904 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:53.104880    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 21:59:53.269204 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:53.269085    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-th2bd" podStartSLOduration=65.894828551 podStartE2EDuration="1m10.269059768s" totalImagesPullingTime="4.374231217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:53.11992509 +0000 UTC m=+70.128854635" watchObservedRunningTime="2026-09-17 21:59:53.269059768 +0000 UTC m=+70.277989309"
Sep 17 21:59:53.269414 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:53.269373    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c9h8t"]
Sep 17 21:59:53.272744 ip-10-0-142-197 kubenswrapper[2567]: W0917 21:59:53.272716    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0374325e_9608_4e5b_9551_0f226154fa74.slice/crio-38fb3690735b4306feb8f3c46e4d62c66058f45f69fe33022f6e62052800340f WatchSource:0}: Error finding container 38fb3690735b4306feb8f3c46e4d62c66058f45f69fe33022f6e62052800340f: Status 404 returned error can't find the container with id 38fb3690735b4306feb8f3c46e4d62c66058f45f69fe33022f6e62052800340f
Sep 17 21:59:54.113992 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:54.113954    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-58crn" event={"ID":"c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673","Type":"ContainerStarted","Data":"5e963c34d354ec4ce76a08cc885bed020a7b0ff00dff50022f0abfe2e8d1a0eb"}
Sep 17 21:59:54.115461 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:54.115427    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c9h8t" event={"ID":"0374325e-9608-4e5b-9551-0f226154fa74","Type":"ContainerStarted","Data":"282a5aa1f3419c66863b32da8ff2ebf964c87c02f2480870c701f6af97d9d135"}
Sep 17 21:59:54.115657 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:54.115482    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c9h8t" event={"ID":"0374325e-9608-4e5b-9551-0f226154fa74","Type":"ContainerStarted","Data":"38fb3690735b4306feb8f3c46e4d62c66058f45f69fe33022f6e62052800340f"}
Sep 17 21:59:54.128656 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:54.128575    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-58crn" podStartSLOduration=66.748981932 podStartE2EDuration="1m11.12856085s" totalImagesPullingTime="4.379578918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:54.128506784 +0000 UTC m=+71.137436322" watchObservedRunningTime="2026-09-17 21:59:54.12856085 +0000 UTC m=+71.137490391"
Sep 17 21:59:56.126205 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:56.126177    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk" event={"ID":"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d","Type":"ContainerStarted","Data":"a3778cdc088fa9211cc9e7a233e01d686899dad4503ba4c410e863b2849c399b"}
Sep 17 21:59:56.128271 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:56.128223    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c9h8t" event={"ID":"0374325e-9608-4e5b-9551-0f226154fa74","Type":"ContainerStarted","Data":"90cba1497aef9e4eba1031d80a046af057de5b0b628da2d82e6f5c408abc4ab1"}
Sep 17 21:59:57.134895 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:57.134860    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk" event={"ID":"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d","Type":"ContainerStarted","Data":"ca6b95d14c9dcd2952196e89317b898f451eb9f4bf42c6305f0c5a54ddf410ea"}
Sep 17 21:59:57.135367 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:57.134910    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk" event={"ID":"86e5fd2b-3100-4d2e-a7bd-3622a7834a4d","Type":"ContainerStarted","Data":"0ef45ddd53c3de85cc32fa702b6e6c7fc71fffd03276537c4cbe7542b679520f"}
Sep 17 21:59:57.156375 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:57.156309    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57dc97d7d8-hv8qk" podStartSLOduration=1.884142327 podStartE2EDuration="17.156293497s" totalImagesPullingTime="15.27215117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:57.154679433 +0000 UTC m=+74.163608973" watchObservedRunningTime="2026-09-17 21:59:57.156293497 +0000 UTC m=+74.165223036"
Sep 17 21:59:58.139584 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:58.139545    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c9h8t" event={"ID":"0374325e-9608-4e5b-9551-0f226154fa74","Type":"ContainerStarted","Data":"630e5e3a2a62698e244fc3dd3187b5a7ee4567cfbe00a42411a968c37e84fc63"}
Sep 17 21:59:58.163273 ip-10-0-142-197 kubenswrapper[2567]: I0917 21:59:58.163233    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c9h8t" podStartSLOduration=36.297259189 podStartE2EDuration="40.163220354s" totalImagesPullingTime="3.865961165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:59:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:58.162773357 +0000 UTC m=+75.171702898" watchObservedRunningTime="2026-09-17 21:59:58.163220354 +0000 UTC m=+75.172149898"
Sep 17 22:00:00.174919 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.174895    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 22:00:00.416856 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.416816    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4zf8r"]
Sep 17 22:00:00.419388 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.419367    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.422243 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.422222    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 22:00:00.422921 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.422884    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 22:00:00.422921 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.422910    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 22:00:00.423075 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.422913    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 22:00:00.423634 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.423619    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 22:00:00.424165 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.424150    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hzs49\""
Sep 17 22:00:00.427090 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.427075    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 22:00:00.461898 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.461880    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-wtmp\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.462021 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.461906    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-accelerators-collector-config\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.462021 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.461922    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxkr4\" (UniqueName: \"kubernetes.io/projected/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-kube-api-access-fxkr4\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.462021 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.461953    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-sys\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.462021 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.461990    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.462174 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.462039    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-textfile\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.462174 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.462064    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-metrics-client-ca\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.462174 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.462081    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-tls\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.462174 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.462103    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-root\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.562601 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.562568    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-metrics-client-ca\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.562601 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.562603    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-tls\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.562859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.562625    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-root\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.562859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.562655    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-wtmp\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.562859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.562678    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-accelerators-collector-config\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.562859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.562702    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxkr4\" (UniqueName: \"kubernetes.io/projected/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-kube-api-access-fxkr4\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.562859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.562716    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-root\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.562859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.562732    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-sys\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.562859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.562780    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-sys\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.562859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.562779    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.562859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.562811    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-wtmp\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.562859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.562833    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-textfile\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.563358 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.563179    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-textfile\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.563358 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.563345    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-metrics-client-ca\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.563440 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.563368    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-accelerators-collector-config\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.565089 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.565070    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-tls\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.565308 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.565293    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.586985 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.586965    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxkr4\" (UniqueName: \"kubernetes.io/projected/d5610702-f4cb-4f8d-8a66-fe4d4b77acb8-kube-api-access-fxkr4\") pod \"node-exporter-4zf8r\" (UID: \"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8\") " pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.729775 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:00.729750    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4zf8r"
Sep 17 22:00:00.741478 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:00:00.741438    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5610702_f4cb_4f8d_8a66_fe4d4b77acb8.slice/crio-2e3c78d1f6f35befb0d134204a6895e6410ac8ade0588003f1bc45a801acf449 WatchSource:0}: Error finding container 2e3c78d1f6f35befb0d134204a6895e6410ac8ade0588003f1bc45a801acf449: Status 404 returned error can't find the container with id 2e3c78d1f6f35befb0d134204a6895e6410ac8ade0588003f1bc45a801acf449
Sep 17 22:00:01.149867 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:01.149794    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4zf8r" event={"ID":"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8","Type":"ContainerStarted","Data":"2e3c78d1f6f35befb0d134204a6895e6410ac8ade0588003f1bc45a801acf449"}
Sep 17 22:00:02.052687 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:02.052656    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fdccdf588-rpvrh"
Sep 17 22:00:02.154730 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:02.154686    2567 generic.go:356] "Generic (PLEG): container finished" podID="d5610702-f4cb-4f8d-8a66-fe4d4b77acb8" containerID="1f75e1b58992908fc6764b59fb57f487effb7f525a4fd5fcbd2e0231f82af173" exitCode=0
Sep 17 22:00:02.154904 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:02.154766    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4zf8r" event={"ID":"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8","Type":"ContainerDied","Data":"1f75e1b58992908fc6764b59fb57f487effb7f525a4fd5fcbd2e0231f82af173"}
Sep 17 22:00:03.160480 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:03.160448    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4zf8r" event={"ID":"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8","Type":"ContainerStarted","Data":"0f4fa20e25165edd43189026250bb5b11750ffed3458c8556bdea601def23848"}
Sep 17 22:00:03.160859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:03.160497    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4zf8r" event={"ID":"d5610702-f4cb-4f8d-8a66-fe4d4b77acb8","Type":"ContainerStarted","Data":"4cc291fbbe644e58f0132e0ab6a20180af9e79f113227e76f6a7bf4d507b3c32"}
Sep 17 22:00:03.182966 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:03.182920    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4zf8r" podStartSLOduration=1.916975767 podStartE2EDuration="3.182903651s" totalImagesPullingTime="1.201711601s" totalInitContainerRuntime="64.216283ms" isStatefulPod=true podCreationTimestamp="2026-09-17 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:03.181032414 +0000 UTC m=+80.189961951" watchObservedRunningTime="2026-09-17 22:00:03.182903651 +0000 UTC m=+80.191833192"
Sep 17 22:00:05.189566 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.189502    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79f5f54446-9cprq" podUID="39ecffbe-80f7-449f-9b29-dca2ffe07044" containerName="registry" containerID="cri-o://0fb1eea9fbc4144e2af05b496339161e6205f054378d6d9ae36ee840a005613d" gracePeriod=30
Sep 17 22:00:05.422350 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.422325    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 22:00:05.501543 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.501511    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39ecffbe-80f7-449f-9b29-dca2ffe07044-ca-trust-extracted\") pod \"39ecffbe-80f7-449f-9b29-dca2ffe07044\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") "
Sep 17 22:00:05.501698 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.501562    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39ecffbe-80f7-449f-9b29-dca2ffe07044-installation-pull-secrets\") pod \"39ecffbe-80f7-449f-9b29-dca2ffe07044\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") "
Sep 17 22:00:05.501698 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.501582    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39ecffbe-80f7-449f-9b29-dca2ffe07044-image-registry-private-configuration\") pod \"39ecffbe-80f7-449f-9b29-dca2ffe07044\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") "
Sep 17 22:00:05.501698 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.501603    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-bound-sa-token\") pod \"39ecffbe-80f7-449f-9b29-dca2ffe07044\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") "
Sep 17 22:00:05.501698 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.501644    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2bs8\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-kube-api-access-k2bs8\") pod \"39ecffbe-80f7-449f-9b29-dca2ffe07044\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") "
Sep 17 22:00:05.501698 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.501662    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-certificates\") pod \"39ecffbe-80f7-449f-9b29-dca2ffe07044\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") "
Sep 17 22:00:05.501698 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.501691    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls\") pod \"39ecffbe-80f7-449f-9b29-dca2ffe07044\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") "
Sep 17 22:00:05.501980 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.501712    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39ecffbe-80f7-449f-9b29-dca2ffe07044-trusted-ca\") pod \"39ecffbe-80f7-449f-9b29-dca2ffe07044\" (UID: \"39ecffbe-80f7-449f-9b29-dca2ffe07044\") "
Sep 17 22:00:05.502397 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.502301    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ecffbe-80f7-449f-9b29-dca2ffe07044-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "39ecffbe-80f7-449f-9b29-dca2ffe07044" (UID: "39ecffbe-80f7-449f-9b29-dca2ffe07044"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:05.502397 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.502308    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "39ecffbe-80f7-449f-9b29-dca2ffe07044" (UID: "39ecffbe-80f7-449f-9b29-dca2ffe07044"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:05.504232 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.504191    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "39ecffbe-80f7-449f-9b29-dca2ffe07044" (UID: "39ecffbe-80f7-449f-9b29-dca2ffe07044"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:05.504232 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.504202    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ecffbe-80f7-449f-9b29-dca2ffe07044-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "39ecffbe-80f7-449f-9b29-dca2ffe07044" (UID: "39ecffbe-80f7-449f-9b29-dca2ffe07044"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:05.504403 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.504246    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "39ecffbe-80f7-449f-9b29-dca2ffe07044" (UID: "39ecffbe-80f7-449f-9b29-dca2ffe07044"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:05.504403 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.504304    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ecffbe-80f7-449f-9b29-dca2ffe07044-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "39ecffbe-80f7-449f-9b29-dca2ffe07044" (UID: "39ecffbe-80f7-449f-9b29-dca2ffe07044"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:05.504403 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.504355    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-kube-api-access-k2bs8" (OuterVolumeSpecName: "kube-api-access-k2bs8") pod "39ecffbe-80f7-449f-9b29-dca2ffe07044" (UID: "39ecffbe-80f7-449f-9b29-dca2ffe07044"). InnerVolumeSpecName "kube-api-access-k2bs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:05.511914 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.511891    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ecffbe-80f7-449f-9b29-dca2ffe07044-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "39ecffbe-80f7-449f-9b29-dca2ffe07044" (UID: "39ecffbe-80f7-449f-9b29-dca2ffe07044"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:00:05.602395 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.602337    2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/39ecffbe-80f7-449f-9b29-dca2ffe07044-image-registry-private-configuration\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:00:05.602395 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.602391    2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-bound-sa-token\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:00:05.602395 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.602401    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2bs8\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-kube-api-access-k2bs8\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:00:05.602395 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.602409    2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-certificates\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:00:05.602606 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.602417    2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/39ecffbe-80f7-449f-9b29-dca2ffe07044-registry-tls\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:00:05.602606 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.602425    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39ecffbe-80f7-449f-9b29-dca2ffe07044-trusted-ca\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:00:05.602606 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.602432    2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/39ecffbe-80f7-449f-9b29-dca2ffe07044-ca-trust-extracted\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:00:05.602606 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:05.602439    2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/39ecffbe-80f7-449f-9b29-dca2ffe07044-installation-pull-secrets\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:00:06.170711 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:06.170678    2567 generic.go:356] "Generic (PLEG): container finished" podID="39ecffbe-80f7-449f-9b29-dca2ffe07044" containerID="0fb1eea9fbc4144e2af05b496339161e6205f054378d6d9ae36ee840a005613d" exitCode=0
Sep 17 22:00:06.170890 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:06.170727    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f5f54446-9cprq"
Sep 17 22:00:06.170890 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:06.170759    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f5f54446-9cprq" event={"ID":"39ecffbe-80f7-449f-9b29-dca2ffe07044","Type":"ContainerDied","Data":"0fb1eea9fbc4144e2af05b496339161e6205f054378d6d9ae36ee840a005613d"}
Sep 17 22:00:06.170890 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:06.170810    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f5f54446-9cprq" event={"ID":"39ecffbe-80f7-449f-9b29-dca2ffe07044","Type":"ContainerDied","Data":"36f99ed652eee4f3c22a1330d1775193fe6998c25b43b2d379cdd8330519c0a0"}
Sep 17 22:00:06.170890 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:06.170835    2567 scope.go:117] "RemoveContainer" containerID="0fb1eea9fbc4144e2af05b496339161e6205f054378d6d9ae36ee840a005613d"
Sep 17 22:00:06.178475 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:06.178459    2567 scope.go:117] "RemoveContainer" containerID="0fb1eea9fbc4144e2af05b496339161e6205f054378d6d9ae36ee840a005613d"
Sep 17 22:00:06.178768 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:00:06.178743    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb1eea9fbc4144e2af05b496339161e6205f054378d6d9ae36ee840a005613d\": container with ID starting with 0fb1eea9fbc4144e2af05b496339161e6205f054378d6d9ae36ee840a005613d not found: ID does not exist" containerID="0fb1eea9fbc4144e2af05b496339161e6205f054378d6d9ae36ee840a005613d"
Sep 17 22:00:06.178846 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:06.178781    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb1eea9fbc4144e2af05b496339161e6205f054378d6d9ae36ee840a005613d"} err="failed to get container status \"0fb1eea9fbc4144e2af05b496339161e6205f054378d6d9ae36ee840a005613d\": rpc error: code = NotFound desc = could not find container \"0fb1eea9fbc4144e2af05b496339161e6205f054378d6d9ae36ee840a005613d\": container with ID starting with 0fb1eea9fbc4144e2af05b496339161e6205f054378d6d9ae36ee840a005613d not found: ID does not exist"
Sep 17 22:00:06.189955 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:06.189934    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79f5f54446-9cprq"]
Sep 17 22:00:06.196479 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:06.196441    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79f5f54446-9cprq"]
Sep 17 22:00:07.639625 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:07.639594    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39ecffbe-80f7-449f-9b29-dca2ffe07044" path="/var/lib/kubelet/pods/39ecffbe-80f7-449f-9b29-dca2ffe07044/volumes"
Sep 17 22:00:25.123055 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:25.123028    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-th2bd"
Sep 17 22:00:29.078372 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:00:29.078342    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4d58n_e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7/serve-healthcheck-canary/0.log"
Sep 17 22:03:43.388147 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:03:43.388117    2567 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 22:05:08.940203 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:08.940171    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nj9p6"]
Sep 17 22:05:08.940685 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:08.940445    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39ecffbe-80f7-449f-9b29-dca2ffe07044" containerName="registry"
Sep 17 22:05:08.940685 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:08.940460    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="39ecffbe-80f7-449f-9b29-dca2ffe07044" containerName="registry"
Sep 17 22:05:08.940685 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:08.940543    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="39ecffbe-80f7-449f-9b29-dca2ffe07044" containerName="registry"
Sep 17 22:05:08.942682 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:08.942662    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-nj9p6"
Sep 17 22:05:08.945115 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:08.945088    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 22:05:08.945228 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:08.945149    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 22:05:08.945297 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:08.945222    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 22:05:08.945986 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:08.945968    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-9sx48\""
Sep 17 22:05:08.946104 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:08.946019    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 22:05:08.954552 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:08.954532    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nj9p6"]
Sep 17 22:05:08.983625 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:08.983601    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e6357d14-eb7f-496f-af6b-07cb51cf472f-certificates\") pod \"keda-admission-6c4d864b86-nj9p6\" (UID: \"e6357d14-eb7f-496f-af6b-07cb51cf472f\") " pod="openshift-keda/keda-admission-6c4d864b86-nj9p6"
Sep 17 22:05:08.983722 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:08.983641    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6czr2\" (UniqueName: \"kubernetes.io/projected/e6357d14-eb7f-496f-af6b-07cb51cf472f-kube-api-access-6czr2\") pod \"keda-admission-6c4d864b86-nj9p6\" (UID: \"e6357d14-eb7f-496f-af6b-07cb51cf472f\") " pod="openshift-keda/keda-admission-6c4d864b86-nj9p6"
Sep 17 22:05:09.084926 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:09.084900    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czr2\" (UniqueName: \"kubernetes.io/projected/e6357d14-eb7f-496f-af6b-07cb51cf472f-kube-api-access-6czr2\") pod \"keda-admission-6c4d864b86-nj9p6\" (UID: \"e6357d14-eb7f-496f-af6b-07cb51cf472f\") " pod="openshift-keda/keda-admission-6c4d864b86-nj9p6"
Sep 17 22:05:09.085073 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:09.084949    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e6357d14-eb7f-496f-af6b-07cb51cf472f-certificates\") pod \"keda-admission-6c4d864b86-nj9p6\" (UID: \"e6357d14-eb7f-496f-af6b-07cb51cf472f\") " pod="openshift-keda/keda-admission-6c4d864b86-nj9p6"
Sep 17 22:05:09.085073 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:05:09.085067    2567 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 17 22:05:09.085158 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:05:09.085089    2567 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-nj9p6: secret "keda-admission-webhooks-certs" not found
Sep 17 22:05:09.085158 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:05:09.085156    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6357d14-eb7f-496f-af6b-07cb51cf472f-certificates podName:e6357d14-eb7f-496f-af6b-07cb51cf472f nodeName:}" failed. No retries permitted until 2026-09-17 22:05:09.585136602 +0000 UTC m=+386.594066132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e6357d14-eb7f-496f-af6b-07cb51cf472f-certificates") pod "keda-admission-6c4d864b86-nj9p6" (UID: "e6357d14-eb7f-496f-af6b-07cb51cf472f") : secret "keda-admission-webhooks-certs" not found
Sep 17 22:05:09.094056 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:09.094032    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czr2\" (UniqueName: \"kubernetes.io/projected/e6357d14-eb7f-496f-af6b-07cb51cf472f-kube-api-access-6czr2\") pod \"keda-admission-6c4d864b86-nj9p6\" (UID: \"e6357d14-eb7f-496f-af6b-07cb51cf472f\") " pod="openshift-keda/keda-admission-6c4d864b86-nj9p6"
Sep 17 22:05:09.587641 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:09.587602    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e6357d14-eb7f-496f-af6b-07cb51cf472f-certificates\") pod \"keda-admission-6c4d864b86-nj9p6\" (UID: \"e6357d14-eb7f-496f-af6b-07cb51cf472f\") " pod="openshift-keda/keda-admission-6c4d864b86-nj9p6"
Sep 17 22:05:09.589893 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:09.589868    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e6357d14-eb7f-496f-af6b-07cb51cf472f-certificates\") pod \"keda-admission-6c4d864b86-nj9p6\" (UID: \"e6357d14-eb7f-496f-af6b-07cb51cf472f\") " pod="openshift-keda/keda-admission-6c4d864b86-nj9p6"
Sep 17 22:05:09.852579 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:09.852498    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-nj9p6"
Sep 17 22:05:10.117014 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:10.116927    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-nj9p6"]
Sep 17 22:05:10.119697 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:05:10.119655    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6357d14_eb7f_496f_af6b_07cb51cf472f.slice/crio-3aea7902cbec755e608163625f0704d7abce1fd35d0fb431592db1901192d808 WatchSource:0}: Error finding container 3aea7902cbec755e608163625f0704d7abce1fd35d0fb431592db1901192d808: Status 404 returned error can't find the container with id 3aea7902cbec755e608163625f0704d7abce1fd35d0fb431592db1901192d808
Sep 17 22:05:10.120868 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:10.120851    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:05:10.143091 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:10.143065    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-nj9p6" event={"ID":"e6357d14-eb7f-496f-af6b-07cb51cf472f","Type":"ContainerStarted","Data":"3aea7902cbec755e608163625f0704d7abce1fd35d0fb431592db1901192d808"}
Sep 17 22:05:13.155437 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:13.155402    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-nj9p6" event={"ID":"e6357d14-eb7f-496f-af6b-07cb51cf472f","Type":"ContainerStarted","Data":"8abf6428bbe33546163aee6f207c7476bd4fe80839d43812f571c8a36eb43d07"}
Sep 17 22:05:13.155886 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:13.155531    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-nj9p6"
Sep 17 22:05:13.170836 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:13.170788    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-nj9p6" podStartSLOduration=2.805803439 podStartE2EDuration="5.170771998s" totalImagesPullingTime="2.364968559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:05:13.17018973 +0000 UTC m=+390.179119284" watchObservedRunningTime="2026-09-17 22:05:13.170771998 +0000 UTC m=+390.179701538"
Sep 17 22:05:34.160425 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:05:34.160395    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-nj9p6"
Sep 17 22:06:37.315354 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:37.315318    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bbwzw"]
Sep 17 22:06:37.317238 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:37.317216    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bbwzw"
Sep 17 22:06:37.319490 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:37.319468    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 22:06:37.319615 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:37.319489    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 22:06:37.320401 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:37.320385    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wpfsj\""
Sep 17 22:06:37.327237 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:37.327215    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bbwzw"]
Sep 17 22:06:37.370050 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:37.370016    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26fcf7b7-1d4f-4dac-9971-64832f551ae8-bound-sa-token\") pod \"cert-manager-55b4d954d5-bbwzw\" (UID: \"26fcf7b7-1d4f-4dac-9971-64832f551ae8\") " pod="cert-manager/cert-manager-55b4d954d5-bbwzw"
Sep 17 22:06:37.370158 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:37.370086    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvcrd\" (UniqueName: \"kubernetes.io/projected/26fcf7b7-1d4f-4dac-9971-64832f551ae8-kube-api-access-gvcrd\") pod \"cert-manager-55b4d954d5-bbwzw\" (UID: \"26fcf7b7-1d4f-4dac-9971-64832f551ae8\") " pod="cert-manager/cert-manager-55b4d954d5-bbwzw"
Sep 17 22:06:37.471142 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:37.471118    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcrd\" (UniqueName: \"kubernetes.io/projected/26fcf7b7-1d4f-4dac-9971-64832f551ae8-kube-api-access-gvcrd\") pod \"cert-manager-55b4d954d5-bbwzw\" (UID: \"26fcf7b7-1d4f-4dac-9971-64832f551ae8\") " pod="cert-manager/cert-manager-55b4d954d5-bbwzw"
Sep 17 22:06:37.471266 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:37.471160    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26fcf7b7-1d4f-4dac-9971-64832f551ae8-bound-sa-token\") pod \"cert-manager-55b4d954d5-bbwzw\" (UID: \"26fcf7b7-1d4f-4dac-9971-64832f551ae8\") " pod="cert-manager/cert-manager-55b4d954d5-bbwzw"
Sep 17 22:06:37.479645 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:37.479610    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26fcf7b7-1d4f-4dac-9971-64832f551ae8-bound-sa-token\") pod \"cert-manager-55b4d954d5-bbwzw\" (UID: \"26fcf7b7-1d4f-4dac-9971-64832f551ae8\") " pod="cert-manager/cert-manager-55b4d954d5-bbwzw"
Sep 17 22:06:37.479751 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:37.479737    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvcrd\" (UniqueName: \"kubernetes.io/projected/26fcf7b7-1d4f-4dac-9971-64832f551ae8-kube-api-access-gvcrd\") pod \"cert-manager-55b4d954d5-bbwzw\" (UID: \"26fcf7b7-1d4f-4dac-9971-64832f551ae8\") " pod="cert-manager/cert-manager-55b4d954d5-bbwzw"
Sep 17 22:06:37.625767 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:37.625705    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bbwzw"
Sep 17 22:06:37.891159 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:37.891069    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bbwzw"]
Sep 17 22:06:37.894922 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:06:37.894886    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26fcf7b7_1d4f_4dac_9971_64832f551ae8.slice/crio-0a93baa04fa66c7056757bac688158143f3d34874a01c81010e6c4bbd0f4cdff WatchSource:0}: Error finding container 0a93baa04fa66c7056757bac688158143f3d34874a01c81010e6c4bbd0f4cdff: Status 404 returned error can't find the container with id 0a93baa04fa66c7056757bac688158143f3d34874a01c81010e6c4bbd0f4cdff
Sep 17 22:06:38.431333 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:38.431300    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bbwzw" event={"ID":"26fcf7b7-1d4f-4dac-9971-64832f551ae8","Type":"ContainerStarted","Data":"0a93baa04fa66c7056757bac688158143f3d34874a01c81010e6c4bbd0f4cdff"}
Sep 17 22:06:41.446959 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:41.446925    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bbwzw" event={"ID":"26fcf7b7-1d4f-4dac-9971-64832f551ae8","Type":"ContainerStarted","Data":"b08978f627f6f7c26ee7acea288f4b3c91da2c58ed799ed8e258fac8736d60c8"}
Sep 17 22:06:41.462218 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:41.462176    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bbwzw" podStartSLOduration=1.393701577 podStartE2EDuration="4.462163926s" totalImagesPullingTime="3.068462349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:06:41.461115006 +0000 UTC m=+478.470044538" watchObservedRunningTime="2026-09-17 22:06:41.462163926 +0000 UTC m=+478.471093468"
Sep 17 22:06:48.912590 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:48.912556    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"]
Sep 17 22:06:48.914563 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:48.914549    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:48.917791 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:48.917770    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 22:06:48.917893 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:48.917815    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 22:06:48.917954 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:48.917874    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 22:06:48.917954 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:48.917915    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zt8ks\""
Sep 17 22:06:48.918092 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:48.917967    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 22:06:48.918148 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:48.918094    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 22:06:48.924491 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:48.924469    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"]
Sep 17 22:06:48.952257 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:48.952230    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b14d03e-83a1-41d5-a8ff-c8d22a32b48b-cert\") pod \"lws-controller-manager-65d86c4f4c-v6q7m\" (UID: \"0b14d03e-83a1-41d5-a8ff-c8d22a32b48b\") " pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:48.952354 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:48.952262    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jszmq\" (UniqueName: \"kubernetes.io/projected/0b14d03e-83a1-41d5-a8ff-c8d22a32b48b-kube-api-access-jszmq\") pod \"lws-controller-manager-65d86c4f4c-v6q7m\" (UID: \"0b14d03e-83a1-41d5-a8ff-c8d22a32b48b\") " pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:48.952354 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:48.952281    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b14d03e-83a1-41d5-a8ff-c8d22a32b48b-metrics-cert\") pod \"lws-controller-manager-65d86c4f4c-v6q7m\" (UID: \"0b14d03e-83a1-41d5-a8ff-c8d22a32b48b\") " pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:48.952354 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:48.952308    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b14d03e-83a1-41d5-a8ff-c8d22a32b48b-manager-config\") pod \"lws-controller-manager-65d86c4f4c-v6q7m\" (UID: \"0b14d03e-83a1-41d5-a8ff-c8d22a32b48b\") " pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:49.053195 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:49.053169    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b14d03e-83a1-41d5-a8ff-c8d22a32b48b-cert\") pod \"lws-controller-manager-65d86c4f4c-v6q7m\" (UID: \"0b14d03e-83a1-41d5-a8ff-c8d22a32b48b\") " pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:49.053312 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:49.053199    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jszmq\" (UniqueName: \"kubernetes.io/projected/0b14d03e-83a1-41d5-a8ff-c8d22a32b48b-kube-api-access-jszmq\") pod \"lws-controller-manager-65d86c4f4c-v6q7m\" (UID: \"0b14d03e-83a1-41d5-a8ff-c8d22a32b48b\") " pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:49.053312 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:49.053218    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b14d03e-83a1-41d5-a8ff-c8d22a32b48b-metrics-cert\") pod \"lws-controller-manager-65d86c4f4c-v6q7m\" (UID: \"0b14d03e-83a1-41d5-a8ff-c8d22a32b48b\") " pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:49.053312 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:49.053262    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b14d03e-83a1-41d5-a8ff-c8d22a32b48b-manager-config\") pod \"lws-controller-manager-65d86c4f4c-v6q7m\" (UID: \"0b14d03e-83a1-41d5-a8ff-c8d22a32b48b\") " pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:49.053935 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:49.053912    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0b14d03e-83a1-41d5-a8ff-c8d22a32b48b-manager-config\") pod \"lws-controller-manager-65d86c4f4c-v6q7m\" (UID: \"0b14d03e-83a1-41d5-a8ff-c8d22a32b48b\") " pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:49.055765 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:49.055745    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b14d03e-83a1-41d5-a8ff-c8d22a32b48b-metrics-cert\") pod \"lws-controller-manager-65d86c4f4c-v6q7m\" (UID: \"0b14d03e-83a1-41d5-a8ff-c8d22a32b48b\") " pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:49.055870 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:49.055803    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b14d03e-83a1-41d5-a8ff-c8d22a32b48b-cert\") pod \"lws-controller-manager-65d86c4f4c-v6q7m\" (UID: \"0b14d03e-83a1-41d5-a8ff-c8d22a32b48b\") " pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:49.065803 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:49.065774    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jszmq\" (UniqueName: \"kubernetes.io/projected/0b14d03e-83a1-41d5-a8ff-c8d22a32b48b-kube-api-access-jszmq\") pod \"lws-controller-manager-65d86c4f4c-v6q7m\" (UID: \"0b14d03e-83a1-41d5-a8ff-c8d22a32b48b\") " pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:49.223693 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:49.223616    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:49.490784 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:49.490702    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"]
Sep 17 22:06:49.492823 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:06:49.492801    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b14d03e_83a1_41d5_a8ff_c8d22a32b48b.slice/crio-d8d9c2158b810894cfcd646cfc7179de3b217439ccfe34e7e117765286086f07 WatchSource:0}: Error finding container d8d9c2158b810894cfcd646cfc7179de3b217439ccfe34e7e117765286086f07: Status 404 returned error can't find the container with id d8d9c2158b810894cfcd646cfc7179de3b217439ccfe34e7e117765286086f07
Sep 17 22:06:50.478748 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:50.478709    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m" event={"ID":"0b14d03e-83a1-41d5-a8ff-c8d22a32b48b","Type":"ContainerStarted","Data":"d8d9c2158b810894cfcd646cfc7179de3b217439ccfe34e7e117765286086f07"}
Sep 17 22:06:52.489577 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:52.489522    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m" event={"ID":"0b14d03e-83a1-41d5-a8ff-c8d22a32b48b","Type":"ContainerStarted","Data":"41f8c1187bae5433e54fb6c79ac22fb5c6481ca644fde60e48d0e630c53c5551"}
Sep 17 22:06:52.489927 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:52.489598    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:06:52.506848 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:06:52.506799    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m" podStartSLOduration=2.253571957 podStartE2EDuration="4.506785884s" totalImagesPullingTime="2.253213927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:06:52.505762461 +0000 UTC m=+489.514692004" watchObservedRunningTime="2026-09-17 22:06:52.506785884 +0000 UTC m=+489.515715423"
Sep 17 22:07:03.495260 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:03.495231    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-65d86c4f4c-v6q7m"
Sep 17 22:07:28.383531 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.383496    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h"]
Sep 17 22:07:28.388650 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.388632    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h"
Sep 17 22:07:28.391146 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.390985    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 22:07:28.391146 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.391016    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 22:07:28.391146 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.391074    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 22:07:28.391962 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.391943    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-z2qkm\""
Sep 17 22:07:28.392043 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.391973    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 22:07:28.394422 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.394400    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h"]
Sep 17 22:07:28.529688 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.529662    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xz29h\" (UID: \"62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h"
Sep 17 22:07:28.529806 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.529697    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkhkt\" (UniqueName: \"kubernetes.io/projected/62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce-kube-api-access-fkhkt\") pod \"kuadrant-console-plugin-fbdd4f445-xz29h\" (UID: \"62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h"
Sep 17 22:07:28.529806 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.529717    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xz29h\" (UID: \"62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h"
Sep 17 22:07:28.630486 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.630461    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xz29h\" (UID: \"62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h"
Sep 17 22:07:28.630605 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.630497    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkhkt\" (UniqueName: \"kubernetes.io/projected/62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce-kube-api-access-fkhkt\") pod \"kuadrant-console-plugin-fbdd4f445-xz29h\" (UID: \"62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h"
Sep 17 22:07:28.630605 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.630514    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xz29h\" (UID: \"62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h"
Sep 17 22:07:28.631147 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.631130    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xz29h\" (UID: \"62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h"
Sep 17 22:07:28.632823 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.632802    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xz29h\" (UID: \"62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h"
Sep 17 22:07:28.637915 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.637866    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkhkt\" (UniqueName: \"kubernetes.io/projected/62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce-kube-api-access-fkhkt\") pod \"kuadrant-console-plugin-fbdd4f445-xz29h\" (UID: \"62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h"
Sep 17 22:07:28.698832 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.698806    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h"
Sep 17 22:07:28.967355 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:28.967277    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h"]
Sep 17 22:07:28.970143 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:07:28.970117    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62ffd6d4_5733_4e8a_bba5_4ac1364ee3ce.slice/crio-82827c460d53ddd06a26d944e7149d1b85ff826b01045e91d72397c7b99fe0b5 WatchSource:0}: Error finding container 82827c460d53ddd06a26d944e7149d1b85ff826b01045e91d72397c7b99fe0b5: Status 404 returned error can't find the container with id 82827c460d53ddd06a26d944e7149d1b85ff826b01045e91d72397c7b99fe0b5
Sep 17 22:07:29.610275 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:29.610242    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h" event={"ID":"62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce","Type":"ContainerStarted","Data":"82827c460d53ddd06a26d944e7149d1b85ff826b01045e91d72397c7b99fe0b5"}
Sep 17 22:07:33.627913 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:33.627826    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h" event={"ID":"62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce","Type":"ContainerStarted","Data":"1ad41b892fb33cfe0aa7415abaebd073394c4842550db12a9415d0c22dfef78e"}
Sep 17 22:07:33.645162 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:07:33.645117    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xz29h" podStartSLOduration=1.282755139 podStartE2EDuration="5.645101246s" totalImagesPullingTime="4.362346107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:07:33.64379848 +0000 UTC m=+530.652728022" watchObservedRunningTime="2026-09-17 22:07:33.645101246 +0000 UTC m=+530.654030789"
Sep 17 22:08:09.580689 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:09.580659    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-f2p64"]
Sep 17 22:08:09.582680 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:09.582661    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-f2p64"
Sep 17 22:08:09.585007 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:09.584973    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nj2t6\""
Sep 17 22:08:09.591142 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:09.591103    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-f2p64"]
Sep 17 22:08:09.619157 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:09.619137    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74mdw\" (UniqueName: \"kubernetes.io/projected/40955ffc-ed40-4c4c-96e6-1a332ec6fa6b-kube-api-access-74mdw\") pod \"authorino-7bcfd8f9cd-f2p64\" (UID: \"40955ffc-ed40-4c4c-96e6-1a332ec6fa6b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-f2p64"
Sep 17 22:08:09.720020 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:09.719982    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74mdw\" (UniqueName: \"kubernetes.io/projected/40955ffc-ed40-4c4c-96e6-1a332ec6fa6b-kube-api-access-74mdw\") pod \"authorino-7bcfd8f9cd-f2p64\" (UID: \"40955ffc-ed40-4c4c-96e6-1a332ec6fa6b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-f2p64"
Sep 17 22:08:09.732555 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:09.732537    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74mdw\" (UniqueName: \"kubernetes.io/projected/40955ffc-ed40-4c4c-96e6-1a332ec6fa6b-kube-api-access-74mdw\") pod \"authorino-7bcfd8f9cd-f2p64\" (UID: \"40955ffc-ed40-4c4c-96e6-1a332ec6fa6b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-f2p64"
Sep 17 22:08:09.753660 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:09.753636    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-j7jtc"]
Sep 17 22:08:09.755566 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:09.755552    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-j7jtc"
Sep 17 22:08:09.765820 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:09.765795    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j7jtc"]
Sep 17 22:08:09.821110 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:09.821087    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6sjt\" (UniqueName: \"kubernetes.io/projected/21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9-kube-api-access-g6sjt\") pod \"authorino-59c974bd48-j7jtc\" (UID: \"21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9\") " pod="kuadrant-system/authorino-59c974bd48-j7jtc"
Sep 17 22:08:09.891886 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:09.891838    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-f2p64"
Sep 17 22:08:09.921913 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:09.921884    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6sjt\" (UniqueName: \"kubernetes.io/projected/21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9-kube-api-access-g6sjt\") pod \"authorino-59c974bd48-j7jtc\" (UID: \"21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9\") " pod="kuadrant-system/authorino-59c974bd48-j7jtc"
Sep 17 22:08:09.930781 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:09.930748    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6sjt\" (UniqueName: \"kubernetes.io/projected/21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9-kube-api-access-g6sjt\") pod \"authorino-59c974bd48-j7jtc\" (UID: \"21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9\") " pod="kuadrant-system/authorino-59c974bd48-j7jtc"
Sep 17 22:08:10.064576 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:10.064547    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-j7jtc"
Sep 17 22:08:10.157873 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:10.157810    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-f2p64"]
Sep 17 22:08:10.160718 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:08:10.160691    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40955ffc_ed40_4c4c_96e6_1a332ec6fa6b.slice/crio-afd515c873d7040ee44568071b58e05316567dff7d13ef1071adb4a335e1acf2 WatchSource:0}: Error finding container afd515c873d7040ee44568071b58e05316567dff7d13ef1071adb4a335e1acf2: Status 404 returned error can't find the container with id afd515c873d7040ee44568071b58e05316567dff7d13ef1071adb4a335e1acf2
Sep 17 22:08:10.325890 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:10.325773    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j7jtc"]
Sep 17 22:08:10.328211 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:08:10.328191    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21e1e149_82f8_4e7d_8e7c_d48af6fe5cf9.slice/crio-f41940e0d70a4a2242f2589dd9dda80eeeff124f361543e1de4b11590aab6fba WatchSource:0}: Error finding container f41940e0d70a4a2242f2589dd9dda80eeeff124f361543e1de4b11590aab6fba: Status 404 returned error can't find the container with id f41940e0d70a4a2242f2589dd9dda80eeeff124f361543e1de4b11590aab6fba
Sep 17 22:08:10.752366 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:10.752334    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-j7jtc" event={"ID":"21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9","Type":"ContainerStarted","Data":"f41940e0d70a4a2242f2589dd9dda80eeeff124f361543e1de4b11590aab6fba"}
Sep 17 22:08:10.753450 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:10.753421    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-f2p64" event={"ID":"40955ffc-ed40-4c4c-96e6-1a332ec6fa6b","Type":"ContainerStarted","Data":"afd515c873d7040ee44568071b58e05316567dff7d13ef1071adb4a335e1acf2"}
Sep 17 22:08:12.762767 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:12.762728    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-f2p64" event={"ID":"40955ffc-ed40-4c4c-96e6-1a332ec6fa6b","Type":"ContainerStarted","Data":"afc0abfeea962d0ea602a71a187245d4e0f6311b2853975bc7e2eedbb337b8b4"}
Sep 17 22:08:12.764315 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:12.764289    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-j7jtc" event={"ID":"21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9","Type":"ContainerStarted","Data":"1c9366f9935b7644ec59eacf275e75455fbdce44662c8eac8ef03085aba7e936"}
Sep 17 22:08:12.778363 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:12.778325    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-f2p64" podStartSLOduration=2.012199248 podStartE2EDuration="3.778309921s" totalImagesPullingTime="1.766110673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:08:12.777281347 +0000 UTC m=+569.786210889" watchObservedRunningTime="2026-09-17 22:08:12.778309921 +0000 UTC m=+569.787239463"
Sep 17 22:08:12.795287 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:12.795234    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-j7jtc" podStartSLOduration=2.201274121 podStartE2EDuration="3.79522263s" totalImagesPullingTime="1.593948509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:08:12.793982312 +0000 UTC m=+569.802911854" watchObservedRunningTime="2026-09-17 22:08:12.79522263 +0000 UTC m=+569.804152170"
Sep 17 22:08:12.821403 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:12.821377    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-f2p64"]
Sep 17 22:08:14.772637 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:14.772598    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-f2p64" podUID="40955ffc-ed40-4c4c-96e6-1a332ec6fa6b" containerName="authorino" containerID="cri-o://afc0abfeea962d0ea602a71a187245d4e0f6311b2853975bc7e2eedbb337b8b4" gracePeriod=30
Sep 17 22:08:15.020330 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:15.020309    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-f2p64"
Sep 17 22:08:15.053940 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:15.053885    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74mdw\" (UniqueName: \"kubernetes.io/projected/40955ffc-ed40-4c4c-96e6-1a332ec6fa6b-kube-api-access-74mdw\") pod \"40955ffc-ed40-4c4c-96e6-1a332ec6fa6b\" (UID: \"40955ffc-ed40-4c4c-96e6-1a332ec6fa6b\") "
Sep 17 22:08:15.055809 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:15.055785    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40955ffc-ed40-4c4c-96e6-1a332ec6fa6b-kube-api-access-74mdw" (OuterVolumeSpecName: "kube-api-access-74mdw") pod "40955ffc-ed40-4c4c-96e6-1a332ec6fa6b" (UID: "40955ffc-ed40-4c4c-96e6-1a332ec6fa6b"). InnerVolumeSpecName "kube-api-access-74mdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:08:15.154965 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:15.154942    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74mdw\" (UniqueName: \"kubernetes.io/projected/40955ffc-ed40-4c4c-96e6-1a332ec6fa6b-kube-api-access-74mdw\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:08:15.776542 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:15.776514    2567 generic.go:356] "Generic (PLEG): container finished" podID="40955ffc-ed40-4c4c-96e6-1a332ec6fa6b" containerID="afc0abfeea962d0ea602a71a187245d4e0f6311b2853975bc7e2eedbb337b8b4" exitCode=0
Sep 17 22:08:15.776874 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:15.776551    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-f2p64"
Sep 17 22:08:15.776874 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:15.776567    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-f2p64" event={"ID":"40955ffc-ed40-4c4c-96e6-1a332ec6fa6b","Type":"ContainerDied","Data":"afc0abfeea962d0ea602a71a187245d4e0f6311b2853975bc7e2eedbb337b8b4"}
Sep 17 22:08:15.776874 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:15.776597    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-f2p64" event={"ID":"40955ffc-ed40-4c4c-96e6-1a332ec6fa6b","Type":"ContainerDied","Data":"afd515c873d7040ee44568071b58e05316567dff7d13ef1071adb4a335e1acf2"}
Sep 17 22:08:15.776874 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:15.776613    2567 scope.go:117] "RemoveContainer" containerID="afc0abfeea962d0ea602a71a187245d4e0f6311b2853975bc7e2eedbb337b8b4"
Sep 17 22:08:15.784140 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:15.784123    2567 scope.go:117] "RemoveContainer" containerID="afc0abfeea962d0ea602a71a187245d4e0f6311b2853975bc7e2eedbb337b8b4"
Sep 17 22:08:15.784371 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:08:15.784354    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc0abfeea962d0ea602a71a187245d4e0f6311b2853975bc7e2eedbb337b8b4\": container with ID starting with afc0abfeea962d0ea602a71a187245d4e0f6311b2853975bc7e2eedbb337b8b4 not found: ID does not exist" containerID="afc0abfeea962d0ea602a71a187245d4e0f6311b2853975bc7e2eedbb337b8b4"
Sep 17 22:08:15.784414 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:15.784380    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc0abfeea962d0ea602a71a187245d4e0f6311b2853975bc7e2eedbb337b8b4"} err="failed to get container status \"afc0abfeea962d0ea602a71a187245d4e0f6311b2853975bc7e2eedbb337b8b4\": rpc error: code = NotFound desc = could not find container \"afc0abfeea962d0ea602a71a187245d4e0f6311b2853975bc7e2eedbb337b8b4\": container with ID starting with afc0abfeea962d0ea602a71a187245d4e0f6311b2853975bc7e2eedbb337b8b4 not found: ID does not exist"
Sep 17 22:08:15.796307 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:15.796288    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-f2p64"]
Sep 17 22:08:15.808838 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:15.806184    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-f2p64"]
Sep 17 22:08:17.639561 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:17.639532    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40955ffc-ed40-4c4c-96e6-1a332ec6fa6b" path="/var/lib/kubelet/pods/40955ffc-ed40-4c4c-96e6-1a332ec6fa6b/volumes"
Sep 17 22:08:32.331080 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.331043    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-pkz4h"]
Sep 17 22:08:32.331522 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.331401    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40955ffc-ed40-4c4c-96e6-1a332ec6fa6b" containerName="authorino"
Sep 17 22:08:32.331522 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.331418    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="40955ffc-ed40-4c4c-96e6-1a332ec6fa6b" containerName="authorino"
Sep 17 22:08:32.331522 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.331479    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="40955ffc-ed40-4c4c-96e6-1a332ec6fa6b" containerName="authorino"
Sep 17 22:08:32.335578 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.335557    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-pkz4h"
Sep 17 22:08:32.337757 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.337731    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 22:08:32.340781 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.340754    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-pkz4h"]
Sep 17 22:08:32.366143 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.366119    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqbwc\" (UniqueName: \"kubernetes.io/projected/5329a7fa-75d7-4297-9c19-912d2496ceb2-kube-api-access-kqbwc\") pod \"authorino-6d9898bd99-pkz4h\" (UID: \"5329a7fa-75d7-4297-9c19-912d2496ceb2\") " pod="kuadrant-system/authorino-6d9898bd99-pkz4h"
Sep 17 22:08:32.366216 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.366156    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5329a7fa-75d7-4297-9c19-912d2496ceb2-tls-cert\") pod \"authorino-6d9898bd99-pkz4h\" (UID: \"5329a7fa-75d7-4297-9c19-912d2496ceb2\") " pod="kuadrant-system/authorino-6d9898bd99-pkz4h"
Sep 17 22:08:32.466915 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.466889    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqbwc\" (UniqueName: \"kubernetes.io/projected/5329a7fa-75d7-4297-9c19-912d2496ceb2-kube-api-access-kqbwc\") pod \"authorino-6d9898bd99-pkz4h\" (UID: \"5329a7fa-75d7-4297-9c19-912d2496ceb2\") " pod="kuadrant-system/authorino-6d9898bd99-pkz4h"
Sep 17 22:08:32.467025 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.466930    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5329a7fa-75d7-4297-9c19-912d2496ceb2-tls-cert\") pod \"authorino-6d9898bd99-pkz4h\" (UID: \"5329a7fa-75d7-4297-9c19-912d2496ceb2\") " pod="kuadrant-system/authorino-6d9898bd99-pkz4h"
Sep 17 22:08:32.469191 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.469174    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5329a7fa-75d7-4297-9c19-912d2496ceb2-tls-cert\") pod \"authorino-6d9898bd99-pkz4h\" (UID: \"5329a7fa-75d7-4297-9c19-912d2496ceb2\") " pod="kuadrant-system/authorino-6d9898bd99-pkz4h"
Sep 17 22:08:32.475283 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.475263    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqbwc\" (UniqueName: \"kubernetes.io/projected/5329a7fa-75d7-4297-9c19-912d2496ceb2-kube-api-access-kqbwc\") pod \"authorino-6d9898bd99-pkz4h\" (UID: \"5329a7fa-75d7-4297-9c19-912d2496ceb2\") " pod="kuadrant-system/authorino-6d9898bd99-pkz4h"
Sep 17 22:08:32.645438 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.645378    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-pkz4h"
Sep 17 22:08:32.910341 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:32.910268    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-pkz4h"]
Sep 17 22:08:32.913476 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:08:32.913451    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5329a7fa_75d7_4297_9c19_912d2496ceb2.slice/crio-19705d708742176430c3d976a777763d8551ce7296722d049db8549c86f8dd2c WatchSource:0}: Error finding container 19705d708742176430c3d976a777763d8551ce7296722d049db8549c86f8dd2c: Status 404 returned error can't find the container with id 19705d708742176430c3d976a777763d8551ce7296722d049db8549c86f8dd2c
Sep 17 22:08:33.836787 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:33.836746    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-pkz4h" event={"ID":"5329a7fa-75d7-4297-9c19-912d2496ceb2","Type":"ContainerStarted","Data":"3f91c18642c20163c13355614a4548e9a7e07903b957e135bc359fe49d86796b"}
Sep 17 22:08:33.836787 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:33.836793    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-pkz4h" event={"ID":"5329a7fa-75d7-4297-9c19-912d2496ceb2","Type":"ContainerStarted","Data":"19705d708742176430c3d976a777763d8551ce7296722d049db8549c86f8dd2c"}
Sep 17 22:08:33.852476 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:33.852421    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-pkz4h" podStartSLOduration=1.076406045 podStartE2EDuration="1.85240251s" totalImagesPullingTime="775.996465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:08:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:08:33.852389128 +0000 UTC m=+590.861318670" watchObservedRunningTime="2026-09-17 22:08:33.85240251 +0000 UTC m=+590.861332066"
Sep 17 22:08:33.881155 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:33.881125    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j7jtc"]
Sep 17 22:08:33.881375 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:33.881352    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-j7jtc" podUID="21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9" containerName="authorino" containerID="cri-o://1c9366f9935b7644ec59eacf275e75455fbdce44662c8eac8ef03085aba7e936" gracePeriod=30
Sep 17 22:08:34.104784 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:34.104764    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-j7jtc"
Sep 17 22:08:34.181556 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:34.181532    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6sjt\" (UniqueName: \"kubernetes.io/projected/21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9-kube-api-access-g6sjt\") pod \"21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9\" (UID: \"21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9\") "
Sep 17 22:08:34.183448 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:34.183425    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9-kube-api-access-g6sjt" (OuterVolumeSpecName: "kube-api-access-g6sjt") pod "21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9" (UID: "21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9"). InnerVolumeSpecName "kube-api-access-g6sjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:08:34.282963 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:34.282902    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6sjt\" (UniqueName: \"kubernetes.io/projected/21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9-kube-api-access-g6sjt\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:08:34.842257 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:34.842230    2567 generic.go:356] "Generic (PLEG): container finished" podID="21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9" containerID="1c9366f9935b7644ec59eacf275e75455fbdce44662c8eac8ef03085aba7e936" exitCode=0
Sep 17 22:08:34.842621 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:34.842270    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-j7jtc"
Sep 17 22:08:34.842621 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:34.842296    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-j7jtc" event={"ID":"21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9","Type":"ContainerDied","Data":"1c9366f9935b7644ec59eacf275e75455fbdce44662c8eac8ef03085aba7e936"}
Sep 17 22:08:34.842621 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:34.842328    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-j7jtc" event={"ID":"21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9","Type":"ContainerDied","Data":"f41940e0d70a4a2242f2589dd9dda80eeeff124f361543e1de4b11590aab6fba"}
Sep 17 22:08:34.842621 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:34.842343    2567 scope.go:117] "RemoveContainer" containerID="1c9366f9935b7644ec59eacf275e75455fbdce44662c8eac8ef03085aba7e936"
Sep 17 22:08:34.850749 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:34.850735    2567 scope.go:117] "RemoveContainer" containerID="1c9366f9935b7644ec59eacf275e75455fbdce44662c8eac8ef03085aba7e936"
Sep 17 22:08:34.850988 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:08:34.850975    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9366f9935b7644ec59eacf275e75455fbdce44662c8eac8ef03085aba7e936\": container with ID starting with 1c9366f9935b7644ec59eacf275e75455fbdce44662c8eac8ef03085aba7e936 not found: ID does not exist" containerID="1c9366f9935b7644ec59eacf275e75455fbdce44662c8eac8ef03085aba7e936"
Sep 17 22:08:34.851043 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:34.851012    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9366f9935b7644ec59eacf275e75455fbdce44662c8eac8ef03085aba7e936"} err="failed to get container status \"1c9366f9935b7644ec59eacf275e75455fbdce44662c8eac8ef03085aba7e936\": rpc error: code = NotFound desc = could not find container \"1c9366f9935b7644ec59eacf275e75455fbdce44662c8eac8ef03085aba7e936\": container with ID starting with 1c9366f9935b7644ec59eacf275e75455fbdce44662c8eac8ef03085aba7e936 not found: ID does not exist"
Sep 17 22:08:34.866539 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:34.866515    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j7jtc"]
Sep 17 22:08:34.874153 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:34.874131    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-j7jtc"]
Sep 17 22:08:35.639859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:08:35.639826    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9" path="/var/lib/kubelet/pods/21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9/volumes"
Sep 17 22:12:29.821738 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:29.821636    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"]
Sep 17 22:12:29.822303 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:29.821935    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9" containerName="authorino"
Sep 17 22:12:29.822303 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:29.821946    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9" containerName="authorino"
Sep 17 22:12:29.822303 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:29.822026    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="21e1e149-82f8-4e7d-8e7c-d48af6fe5cf9" containerName="authorino"
Sep 17 22:12:29.824727 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:29.824707    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"
Sep 17 22:12:29.828161 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:29.828116    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 22:12:29.828277 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:29.828219    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-4s777\""
Sep 17 22:12:29.828340 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:29.828299    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 22:12:29.828340 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:29.828323    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-w4qqf\""
Sep 17 22:12:29.828340 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:29.828327    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 22:12:29.833831 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:29.833815    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"]
Sep 17 22:12:29.963491 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:29.963463    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d11a19c3-0c36-4c98-a0ad-4b750c048bc2-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr\" (UID: \"d11a19c3-0c36-4c98-a0ad-4b750c048bc2\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"
Sep 17 22:12:29.963642 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:29.963515    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jlnz\" (UniqueName: \"kubernetes.io/projected/d11a19c3-0c36-4c98-a0ad-4b750c048bc2-kube-api-access-5jlnz\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr\" (UID: \"d11a19c3-0c36-4c98-a0ad-4b750c048bc2\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"
Sep 17 22:12:30.064767 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:30.064736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d11a19c3-0c36-4c98-a0ad-4b750c048bc2-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr\" (UID: \"d11a19c3-0c36-4c98-a0ad-4b750c048bc2\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"
Sep 17 22:12:30.064907 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:30.064784    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jlnz\" (UniqueName: \"kubernetes.io/projected/d11a19c3-0c36-4c98-a0ad-4b750c048bc2-kube-api-access-5jlnz\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr\" (UID: \"d11a19c3-0c36-4c98-a0ad-4b750c048bc2\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"
Sep 17 22:12:30.067178 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:30.067156    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d11a19c3-0c36-4c98-a0ad-4b750c048bc2-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr\" (UID: \"d11a19c3-0c36-4c98-a0ad-4b750c048bc2\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"
Sep 17 22:12:30.072723 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:30.072672    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jlnz\" (UniqueName: \"kubernetes.io/projected/d11a19c3-0c36-4c98-a0ad-4b750c048bc2-kube-api-access-5jlnz\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr\" (UID: \"d11a19c3-0c36-4c98-a0ad-4b750c048bc2\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"
Sep 17 22:12:30.137134 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:30.137107    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"
Sep 17 22:12:30.406875 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:30.406749    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"]
Sep 17 22:12:30.410500 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:12:30.410467    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11a19c3_0c36_4c98_a0ad_4b750c048bc2.slice/crio-d038b640fc0c799cb4661dd3c0c94c5ebdc82ad8a2fe2a485e3051e6ca9f9c4c WatchSource:0}: Error finding container d038b640fc0c799cb4661dd3c0c94c5ebdc82ad8a2fe2a485e3051e6ca9f9c4c: Status 404 returned error can't find the container with id d038b640fc0c799cb4661dd3c0c94c5ebdc82ad8a2fe2a485e3051e6ca9f9c4c
Sep 17 22:12:30.412542 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:30.412524    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:12:30.664082 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:30.663988    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr" event={"ID":"d11a19c3-0c36-4c98-a0ad-4b750c048bc2","Type":"ContainerStarted","Data":"d038b640fc0c799cb4661dd3c0c94c5ebdc82ad8a2fe2a485e3051e6ca9f9c4c"}
Sep 17 22:12:33.676028 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:33.675979    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr" event={"ID":"d11a19c3-0c36-4c98-a0ad-4b750c048bc2","Type":"ContainerStarted","Data":"4b5a8ec3fbca44676326c09f773c596a64019fb9658bfbe11a62488fb237caca"}
Sep 17 22:12:33.676415 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:33.676116    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"
Sep 17 22:12:33.690916 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:33.690871    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr" podStartSLOduration=2.467359398 podStartE2EDuration="4.69085906s" totalImagesPullingTime="2.223499662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:12:33.689651582 +0000 UTC m=+830.698581124" watchObservedRunningTime="2026-09-17 22:12:33.69085906 +0000 UTC m=+830.699788600"
Sep 17 22:12:54.534833 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.534801    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"]
Sep 17 22:12:54.538218 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.538195    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.541613 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.541591    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 22:12:54.541613 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.541602    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 22:12:54.541769 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.541611    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-tj9m4\""
Sep 17 22:12:54.541769 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.541594    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 22:12:54.547975 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.547956    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"]
Sep 17 22:12:54.625724 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.625697    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.625826 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.625731    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.625826 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.625755    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.625826 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.625789    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xsvz\" (UniqueName: \"kubernetes.io/projected/8532a721-5430-4a68-8639-53d094907a83-kube-api-access-6xsvz\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.625983 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.625861    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.625983 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.625894    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8532a721-5430-4a68-8639-53d094907a83-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.726546 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.726521    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8532a721-5430-4a68-8639-53d094907a83-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.726647 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.726558    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.726647 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.726578    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.726647 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.726600    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.726647 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.726624    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xsvz\" (UniqueName: \"kubernetes.io/projected/8532a721-5430-4a68-8639-53d094907a83-kube-api-access-6xsvz\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.726844 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.726776    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.726940 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.726911    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.727058 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.726955    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.727058 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.726989    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.728768 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.728753    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.729356 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.729339    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8532a721-5430-4a68-8639-53d094907a83-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.737698 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.737680    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xsvz\" (UniqueName: \"kubernetes.io/projected/8532a721-5430-4a68-8639-53d094907a83-kube-api-access-6xsvz\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.784644 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.784619    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"]
Sep 17 22:12:54.788059 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.787983    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"
Sep 17 22:12:54.790179 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.790159    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-bs5v2\""
Sep 17 22:12:54.798978 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.798957    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"]
Sep 17 22:12:54.827128 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.827103    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d146942a-bc18-4566-8968-0be47dbae564-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm\" (UID: \"d146942a-bc18-4566-8968-0be47dbae564\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"
Sep 17 22:12:54.827216 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.827180    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srl4r\" (UniqueName: \"kubernetes.io/projected/d146942a-bc18-4566-8968-0be47dbae564-kube-api-access-srl4r\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm\" (UID: \"d146942a-bc18-4566-8968-0be47dbae564\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"
Sep 17 22:12:54.848105 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.848087    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:12:54.927581 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.927553    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srl4r\" (UniqueName: \"kubernetes.io/projected/d146942a-bc18-4566-8968-0be47dbae564-kube-api-access-srl4r\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm\" (UID: \"d146942a-bc18-4566-8968-0be47dbae564\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"
Sep 17 22:12:54.927685 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.927598    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d146942a-bc18-4566-8968-0be47dbae564-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm\" (UID: \"d146942a-bc18-4566-8968-0be47dbae564\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"
Sep 17 22:12:54.929888 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.929871    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d146942a-bc18-4566-8968-0be47dbae564-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm\" (UID: \"d146942a-bc18-4566-8968-0be47dbae564\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"
Sep 17 22:12:54.935248 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:54.935227    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srl4r\" (UniqueName: \"kubernetes.io/projected/d146942a-bc18-4566-8968-0be47dbae564-kube-api-access-srl4r\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm\" (UID: \"d146942a-bc18-4566-8968-0be47dbae564\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"
Sep 17 22:12:55.098660 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:55.098600    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"
Sep 17 22:12:55.116464 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:55.116440    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"]
Sep 17 22:12:55.118565 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:12:55.118541    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8532a721_5430_4a68_8639_53d094907a83.slice/crio-dd3fd1e5530b894e8d9f0a7eef4821b68221b5efb68228b9263f51b6cf909d44 WatchSource:0}: Error finding container dd3fd1e5530b894e8d9f0a7eef4821b68221b5efb68228b9263f51b6cf909d44: Status 404 returned error can't find the container with id dd3fd1e5530b894e8d9f0a7eef4821b68221b5efb68228b9263f51b6cf909d44
Sep 17 22:12:55.368967 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:55.368940    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"]
Sep 17 22:12:55.405124 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:12:55.405089    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd146942a_bc18_4566_8968_0be47dbae564.slice/crio-95360fc799260d8a48c83696266653f7c25ae78428a8329b169a7cbc859442ff WatchSource:0}: Error finding container 95360fc799260d8a48c83696266653f7c25ae78428a8329b169a7cbc859442ff: Status 404 returned error can't find the container with id 95360fc799260d8a48c83696266653f7c25ae78428a8329b169a7cbc859442ff
Sep 17 22:12:55.758262 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:55.758223    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm" event={"ID":"d146942a-bc18-4566-8968-0be47dbae564","Type":"ContainerStarted","Data":"a360b6d908dc3396a84ef520efba68f11e8d8581715a2730bfdc5d02ad65277a"}
Sep 17 22:12:55.758714 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:55.758327    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm" event={"ID":"d146942a-bc18-4566-8968-0be47dbae564","Type":"ContainerStarted","Data":"95360fc799260d8a48c83696266653f7c25ae78428a8329b169a7cbc859442ff"}
Sep 17 22:12:55.758714 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:55.758358    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"
Sep 17 22:12:55.759705 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:55.759682    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt" event={"ID":"8532a721-5430-4a68-8639-53d094907a83","Type":"ContainerStarted","Data":"dd3fd1e5530b894e8d9f0a7eef4821b68221b5efb68228b9263f51b6cf909d44"}
Sep 17 22:12:55.778287 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:55.777752    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm" podStartSLOduration=1.777730587 podStartE2EDuration="1.777730587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:12:55.77654881 +0000 UTC m=+852.785478354" watchObservedRunningTime="2026-09-17 22:12:55.777730587 +0000 UTC m=+852.786660130"
Sep 17 22:12:56.764665 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:56.764635    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt" event={"ID":"8532a721-5430-4a68-8639-53d094907a83","Type":"ContainerStarted","Data":"b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1"}
Sep 17 22:12:56.780623 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:56.780571    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt" podStartSLOduration=1.356733709 podStartE2EDuration="2.780552689s" totalImagesPullingTime="1.42381898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:12:56.779799529 +0000 UTC m=+853.788729071" watchObservedRunningTime="2026-09-17 22:12:56.780552689 +0000 UTC m=+853.789482238"
Sep 17 22:12:56.848895 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:12:56.848834    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:13:04.682606 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:04.682575    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"
Sep 17 22:13:04.849257 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:04.849224    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:13:04.861248 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:04.861221    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:13:05.805429 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:05.805401    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:13:26.766093 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:26.766065    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"
Sep 17 22:13:28.384043 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.384011    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"]
Sep 17 22:13:28.386014 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.385977    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"]
Sep 17 22:13:28.395954 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:13:28.395918    2567 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:13:28.395954 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:13:28.395954 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:13:28.395954 ip-10-0-142-197 kubenswrapper[2567]:  > containerID="b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1" cmd=["/bin/sleep","15"]
Sep 17 22:13:28.396229 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.395982    2567 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt" message="" err=<
Sep 17 22:13:28.396229 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:13:28.396229 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:13:28.396229 ip-10-0-142-197 kubenswrapper[2567]:  >
Sep 17 22:13:28.396229 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:13:28.396010    2567 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:13:28.396229 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:13:28.396229 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:13:28.396229 ip-10-0-142-197 kubenswrapper[2567]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt" podUID="8532a721-5430-4a68-8639-53d094907a83" containerName="main" containerID="cri-o://b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1"
Sep 17 22:13:28.396229 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.396056    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt" podUID="8532a721-5430-4a68-8639-53d094907a83" containerName="main" containerID="cri-o://b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1" gracePeriod=60
Sep 17 22:13:28.749302 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.749259    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:13:28.869111 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.869081    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-dshm\") pod \"8532a721-5430-4a68-8639-53d094907a83\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") "
Sep 17 22:13:28.869309 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.869128    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xsvz\" (UniqueName: \"kubernetes.io/projected/8532a721-5430-4a68-8639-53d094907a83-kube-api-access-6xsvz\") pod \"8532a721-5430-4a68-8639-53d094907a83\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") "
Sep 17 22:13:28.869309 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.869160    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-home\") pod \"8532a721-5430-4a68-8639-53d094907a83\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") "
Sep 17 22:13:28.869309 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.869184    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-tmp-dir\") pod \"8532a721-5430-4a68-8639-53d094907a83\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") "
Sep 17 22:13:28.869470 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.869318    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8532a721-5430-4a68-8639-53d094907a83-tls-certs\") pod \"8532a721-5430-4a68-8639-53d094907a83\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") "
Sep 17 22:13:28.869470 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.869358    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-model-cache\") pod \"8532a721-5430-4a68-8639-53d094907a83\" (UID: \"8532a721-5430-4a68-8639-53d094907a83\") "
Sep 17 22:13:28.869470 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.869422    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-home" (OuterVolumeSpecName: "home") pod "8532a721-5430-4a68-8639-53d094907a83" (UID: "8532a721-5430-4a68-8639-53d094907a83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:13:28.869470 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.869437    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8532a721-5430-4a68-8639-53d094907a83" (UID: "8532a721-5430-4a68-8639-53d094907a83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:13:28.869644 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.869615    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:13:28.869644 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.869633    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:13:28.869723 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.869656    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-model-cache" (OuterVolumeSpecName: "model-cache") pod "8532a721-5430-4a68-8639-53d094907a83" (UID: "8532a721-5430-4a68-8639-53d094907a83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:13:28.871458 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.871426    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8532a721-5430-4a68-8639-53d094907a83-kube-api-access-6xsvz" (OuterVolumeSpecName: "kube-api-access-6xsvz") pod "8532a721-5430-4a68-8639-53d094907a83" (UID: "8532a721-5430-4a68-8639-53d094907a83"). InnerVolumeSpecName "kube-api-access-6xsvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:13:28.871458 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.871433    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8532a721-5430-4a68-8639-53d094907a83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8532a721-5430-4a68-8639-53d094907a83" (UID: "8532a721-5430-4a68-8639-53d094907a83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:13:28.871592 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.871474    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-dshm" (OuterVolumeSpecName: "dshm") pod "8532a721-5430-4a68-8639-53d094907a83" (UID: "8532a721-5430-4a68-8639-53d094907a83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:13:28.884920 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.884895    2567 generic.go:356] "Generic (PLEG): container finished" podID="8532a721-5430-4a68-8639-53d094907a83" containerID="b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1" exitCode=0
Sep 17 22:13:28.885022 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.884940    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"
Sep 17 22:13:28.885022 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.884949    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt" event={"ID":"8532a721-5430-4a68-8639-53d094907a83","Type":"ContainerDied","Data":"b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1"}
Sep 17 22:13:28.885022 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.885008    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt" event={"ID":"8532a721-5430-4a68-8639-53d094907a83","Type":"ContainerDied","Data":"dd3fd1e5530b894e8d9f0a7eef4821b68221b5efb68228b9263f51b6cf909d44"}
Sep 17 22:13:28.885128 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.885024    2567 scope.go:117] "RemoveContainer" containerID="b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1"
Sep 17 22:13:28.893424 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.893407    2567 scope.go:117] "RemoveContainer" containerID="b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1"
Sep 17 22:13:28.893678 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:13:28.893657    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1\": container with ID starting with b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1 not found: ID does not exist" containerID="b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1"
Sep 17 22:13:28.893757 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.893682    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1"} err="failed to get container status \"b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1\": rpc error: code = NotFound desc = could not find container \"b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1\": container with ID starting with b081494513a18de0d49370b385a25fda6083353971bf82f98cb255ecaebf9fc1 not found: ID does not exist"
Sep 17 22:13:28.909820 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.909749    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"]
Sep 17 22:13:28.910405 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.910378    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-zbkdt"]
Sep 17 22:13:28.970570 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.970533    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:13:28.970570 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.970567    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8532a721-5430-4a68-8639-53d094907a83-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:13:28.970714 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.970576    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xsvz\" (UniqueName: \"kubernetes.io/projected/8532a721-5430-4a68-8639-53d094907a83-kube-api-access-6xsvz\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:13:28.970714 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:28.970586    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8532a721-5430-4a68-8639-53d094907a83-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:13:29.639911 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:29.639877    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8532a721-5430-4a68-8639-53d094907a83" path="/var/lib/kubelet/pods/8532a721-5430-4a68-8639-53d094907a83/volumes"
Sep 17 22:13:30.468023 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:30.467679    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"]
Sep 17 22:13:34.682308 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:34.682273    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr" podUID="d11a19c3-0c36-4c98-a0ad-4b750c048bc2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:13:36.766285 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:36.766202    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm" podUID="d146942a-bc18-4566-8968-0be47dbae564" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:13:39.747945 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.747828    2567 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 17 22:13:39.747945 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:13:39.747865    2567 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm" podUID="d146942a-bc18-4566-8968-0be47dbae564" containerName="main" containerID="cri-o://a360b6d908dc3396a84ef520efba68f11e8d8581715a2730bfdc5d02ad65277a"
Sep 17 22:13:39.747945 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.747917    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm" podUID="d146942a-bc18-4566-8968-0be47dbae564" containerName="main" containerID="cri-o://a360b6d908dc3396a84ef520efba68f11e8d8581715a2730bfdc5d02ad65277a" gracePeriod=49
Sep 17 22:13:39.881309 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.881286    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm_d146942a-bc18-4566-8968-0be47dbae564/main/0.log"
Sep 17 22:13:39.883452 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.883433    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"
Sep 17 22:13:39.926367 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.926343    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm_d146942a-bc18-4566-8968-0be47dbae564/main/0.log"
Sep 17 22:13:39.926516 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.926386    2567 generic.go:356] "Generic (PLEG): container finished" podID="d146942a-bc18-4566-8968-0be47dbae564" containerID="a360b6d908dc3396a84ef520efba68f11e8d8581715a2730bfdc5d02ad65277a" exitCode=1
Sep 17 22:13:39.926516 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.926454    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"
Sep 17 22:13:39.926516 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.926476    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm" event={"ID":"d146942a-bc18-4566-8968-0be47dbae564","Type":"ContainerDied","Data":"a360b6d908dc3396a84ef520efba68f11e8d8581715a2730bfdc5d02ad65277a"}
Sep 17 22:13:39.926516 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.926516    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm" event={"ID":"d146942a-bc18-4566-8968-0be47dbae564","Type":"ContainerDied","Data":"95360fc799260d8a48c83696266653f7c25ae78428a8329b169a7cbc859442ff"}
Sep 17 22:13:39.926676 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.926537    2567 scope.go:117] "RemoveContainer" containerID="a360b6d908dc3396a84ef520efba68f11e8d8581715a2730bfdc5d02ad65277a"
Sep 17 22:13:39.934307 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.934288    2567 scope.go:117] "RemoveContainer" containerID="a360b6d908dc3396a84ef520efba68f11e8d8581715a2730bfdc5d02ad65277a"
Sep 17 22:13:39.934606 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:13:39.934586    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a360b6d908dc3396a84ef520efba68f11e8d8581715a2730bfdc5d02ad65277a\": container with ID starting with a360b6d908dc3396a84ef520efba68f11e8d8581715a2730bfdc5d02ad65277a not found: ID does not exist" containerID="a360b6d908dc3396a84ef520efba68f11e8d8581715a2730bfdc5d02ad65277a"
Sep 17 22:13:39.934678 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.934613    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a360b6d908dc3396a84ef520efba68f11e8d8581715a2730bfdc5d02ad65277a"} err="failed to get container status \"a360b6d908dc3396a84ef520efba68f11e8d8581715a2730bfdc5d02ad65277a\": rpc error: code = NotFound desc = could not find container \"a360b6d908dc3396a84ef520efba68f11e8d8581715a2730bfdc5d02ad65277a\": container with ID starting with a360b6d908dc3396a84ef520efba68f11e8d8581715a2730bfdc5d02ad65277a not found: ID does not exist"
Sep 17 22:13:39.948913 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.948894    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d146942a-bc18-4566-8968-0be47dbae564-tls-certs\") pod \"d146942a-bc18-4566-8968-0be47dbae564\" (UID: \"d146942a-bc18-4566-8968-0be47dbae564\") "
Sep 17 22:13:39.949024 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.948926    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srl4r\" (UniqueName: \"kubernetes.io/projected/d146942a-bc18-4566-8968-0be47dbae564-kube-api-access-srl4r\") pod \"d146942a-bc18-4566-8968-0be47dbae564\" (UID: \"d146942a-bc18-4566-8968-0be47dbae564\") "
Sep 17 22:13:39.950701 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.950676    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d146942a-bc18-4566-8968-0be47dbae564-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d146942a-bc18-4566-8968-0be47dbae564" (UID: "d146942a-bc18-4566-8968-0be47dbae564"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:13:39.950774 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:39.950761    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d146942a-bc18-4566-8968-0be47dbae564-kube-api-access-srl4r" (OuterVolumeSpecName: "kube-api-access-srl4r") pod "d146942a-bc18-4566-8968-0be47dbae564" (UID: "d146942a-bc18-4566-8968-0be47dbae564"). InnerVolumeSpecName "kube-api-access-srl4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:13:40.049983 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:40.049916    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d146942a-bc18-4566-8968-0be47dbae564-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:13:40.049983 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:40.049936    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srl4r\" (UniqueName: \"kubernetes.io/projected/d146942a-bc18-4566-8968-0be47dbae564-kube-api-access-srl4r\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:13:40.246129 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:40.246104    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"]
Sep 17 22:13:40.250110 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:40.250090    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vh5vm"]
Sep 17 22:13:41.639951 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:41.639917    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d146942a-bc18-4566-8968-0be47dbae564" path="/var/lib/kubelet/pods/d146942a-bc18-4566-8968-0be47dbae564/volumes"
Sep 17 22:13:44.682422 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:44.682381    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr" podUID="d11a19c3-0c36-4c98-a0ad-4b750c048bc2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:13:45.489960 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:45.489921    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr" podUID="d11a19c3-0c36-4c98-a0ad-4b750c048bc2" containerName="main" containerID="cri-o://4b5a8ec3fbca44676326c09f773c596a64019fb9658bfbe11a62488fb237caca" gracePeriod=45
Sep 17 22:13:54.681869 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:54.681824    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr" podUID="d11a19c3-0c36-4c98-a0ad-4b750c048bc2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:13:54.682252 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:13:54.681924    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"
Sep 17 22:14:04.681973 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:04.681936    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr" podUID="d11a19c3-0c36-4c98-a0ad-4b750c048bc2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:14:14.682294 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:14.682256    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr" podUID="d11a19c3-0c36-4c98-a0ad-4b750c048bc2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:14:15.727920 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:15.727899    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"
Sep 17 22:14:15.809312 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:15.809218    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d11a19c3-0c36-4c98-a0ad-4b750c048bc2-tls-certs\") pod \"d11a19c3-0c36-4c98-a0ad-4b750c048bc2\" (UID: \"d11a19c3-0c36-4c98-a0ad-4b750c048bc2\") "
Sep 17 22:14:15.809312 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:15.809300    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jlnz\" (UniqueName: \"kubernetes.io/projected/d11a19c3-0c36-4c98-a0ad-4b750c048bc2-kube-api-access-5jlnz\") pod \"d11a19c3-0c36-4c98-a0ad-4b750c048bc2\" (UID: \"d11a19c3-0c36-4c98-a0ad-4b750c048bc2\") "
Sep 17 22:14:15.811374 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:15.811344    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d11a19c3-0c36-4c98-a0ad-4b750c048bc2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d11a19c3-0c36-4c98-a0ad-4b750c048bc2" (UID: "d11a19c3-0c36-4c98-a0ad-4b750c048bc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:14:15.811467 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:15.811348    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11a19c3-0c36-4c98-a0ad-4b750c048bc2-kube-api-access-5jlnz" (OuterVolumeSpecName: "kube-api-access-5jlnz") pod "d11a19c3-0c36-4c98-a0ad-4b750c048bc2" (UID: "d11a19c3-0c36-4c98-a0ad-4b750c048bc2"). InnerVolumeSpecName "kube-api-access-5jlnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:14:15.910132 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:15.910091    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jlnz\" (UniqueName: \"kubernetes.io/projected/d11a19c3-0c36-4c98-a0ad-4b750c048bc2-kube-api-access-5jlnz\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:14:15.910132 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:15.910128    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d11a19c3-0c36-4c98-a0ad-4b750c048bc2-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:14:16.052630 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:16.052594    2567 generic.go:356] "Generic (PLEG): container finished" podID="d11a19c3-0c36-4c98-a0ad-4b750c048bc2" containerID="4b5a8ec3fbca44676326c09f773c596a64019fb9658bfbe11a62488fb237caca" exitCode=0
Sep 17 22:14:16.052794 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:16.052638    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"
Sep 17 22:14:16.052794 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:16.052648    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr" event={"ID":"d11a19c3-0c36-4c98-a0ad-4b750c048bc2","Type":"ContainerDied","Data":"4b5a8ec3fbca44676326c09f773c596a64019fb9658bfbe11a62488fb237caca"}
Sep 17 22:14:16.052794 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:16.052695    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr" event={"ID":"d11a19c3-0c36-4c98-a0ad-4b750c048bc2","Type":"ContainerDied","Data":"d038b640fc0c799cb4661dd3c0c94c5ebdc82ad8a2fe2a485e3051e6ca9f9c4c"}
Sep 17 22:14:16.052794 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:16.052719    2567 scope.go:117] "RemoveContainer" containerID="4b5a8ec3fbca44676326c09f773c596a64019fb9658bfbe11a62488fb237caca"
Sep 17 22:14:16.063774 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:16.063752    2567 scope.go:117] "RemoveContainer" containerID="4b5a8ec3fbca44676326c09f773c596a64019fb9658bfbe11a62488fb237caca"
Sep 17 22:14:16.064089 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:14:16.064068    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5a8ec3fbca44676326c09f773c596a64019fb9658bfbe11a62488fb237caca\": container with ID starting with 4b5a8ec3fbca44676326c09f773c596a64019fb9658bfbe11a62488fb237caca not found: ID does not exist" containerID="4b5a8ec3fbca44676326c09f773c596a64019fb9658bfbe11a62488fb237caca"
Sep 17 22:14:16.064153 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:16.064095    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5a8ec3fbca44676326c09f773c596a64019fb9658bfbe11a62488fb237caca"} err="failed to get container status \"4b5a8ec3fbca44676326c09f773c596a64019fb9658bfbe11a62488fb237caca\": rpc error: code = NotFound desc = could not find container \"4b5a8ec3fbca44676326c09f773c596a64019fb9658bfbe11a62488fb237caca\": container with ID starting with 4b5a8ec3fbca44676326c09f773c596a64019fb9658bfbe11a62488fb237caca not found: ID does not exist"
Sep 17 22:14:16.074280 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:16.074260    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"]
Sep 17 22:14:16.081872 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:16.078691    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-wfpjr"]
Sep 17 22:14:17.639090 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:17.639058    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d11a19c3-0c36-4c98-a0ad-4b750c048bc2" path="/var/lib/kubelet/pods/d11a19c3-0c36-4c98-a0ad-4b750c048bc2/volumes"
Sep 17 22:14:29.135545 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.135517    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"]
Sep 17 22:14:29.135908 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.135771    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d11a19c3-0c36-4c98-a0ad-4b750c048bc2" containerName="main"
Sep 17 22:14:29.135908 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.135782    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d11a19c3-0c36-4c98-a0ad-4b750c048bc2" containerName="main"
Sep 17 22:14:29.135908 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.135791    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d146942a-bc18-4566-8968-0be47dbae564" containerName="main"
Sep 17 22:14:29.135908 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.135796    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d146942a-bc18-4566-8968-0be47dbae564" containerName="main"
Sep 17 22:14:29.135908 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.135805    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8532a721-5430-4a68-8639-53d094907a83" containerName="main"
Sep 17 22:14:29.135908 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.135809    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8532a721-5430-4a68-8639-53d094907a83" containerName="main"
Sep 17 22:14:29.135908 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.135867    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d11a19c3-0c36-4c98-a0ad-4b750c048bc2" containerName="main"
Sep 17 22:14:29.135908 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.135876    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d146942a-bc18-4566-8968-0be47dbae564" containerName="main"
Sep 17 22:14:29.135908 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.135882    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8532a721-5430-4a68-8639-53d094907a83" containerName="main"
Sep 17 22:14:29.140095 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.140078    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:29.142432 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.142403    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-rf542\""
Sep 17 22:14:29.142432 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.142424    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 22:14:29.143342 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.143329    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 22:14:29.143404 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.143393    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-qkgs4\""
Sep 17 22:14:29.143451 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.143432    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 22:14:29.148265 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.148244    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"]
Sep 17 22:14:29.306206 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.306167    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace1cc9-f70e-4814-9542-8efb0b942784-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:29.306206 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.306206    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace1cc9-f70e-4814-9542-8efb0b942784-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:29.306419 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.306280    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thdfj\" (UniqueName: \"kubernetes.io/projected/1ace1cc9-f70e-4814-9542-8efb0b942784-kube-api-access-thdfj\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:29.306419 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.306322    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ace1cc9-f70e-4814-9542-8efb0b942784-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:29.407202 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.407134    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thdfj\" (UniqueName: \"kubernetes.io/projected/1ace1cc9-f70e-4814-9542-8efb0b942784-kube-api-access-thdfj\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:29.407202 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.407169    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ace1cc9-f70e-4814-9542-8efb0b942784-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:29.407389 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.407211    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace1cc9-f70e-4814-9542-8efb0b942784-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:29.407389 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.407246    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace1cc9-f70e-4814-9542-8efb0b942784-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:29.407576 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.407558    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace1cc9-f70e-4814-9542-8efb0b942784-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:29.407642 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.407599    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace1cc9-f70e-4814-9542-8efb0b942784-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:29.409641 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.409619    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ace1cc9-f70e-4814-9542-8efb0b942784-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:29.413956 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.413934    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thdfj\" (UniqueName: \"kubernetes.io/projected/1ace1cc9-f70e-4814-9542-8efb0b942784-kube-api-access-thdfj\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:29.450887 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.450867    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:29.716685 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:29.716539    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"]
Sep 17 22:14:29.719086 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:14:29.719060    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ace1cc9_f70e_4814_9542_8efb0b942784.slice/crio-4479924f229c0ead4efc28623a1cdef3ea385fb7f8f5e89af0dea31964108c03 WatchSource:0}: Error finding container 4479924f229c0ead4efc28623a1cdef3ea385fb7f8f5e89af0dea31964108c03: Status 404 returned error can't find the container with id 4479924f229c0ead4efc28623a1cdef3ea385fb7f8f5e89af0dea31964108c03
Sep 17 22:14:30.103466 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:30.103425    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6" event={"ID":"1ace1cc9-f70e-4814-9542-8efb0b942784","Type":"ContainerStarted","Data":"9a15c5a1eb0456ad452305ff9559ce4580468ff04b787f3b47f8d196ef4aae98"}
Sep 17 22:14:30.103626 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:30.103494    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6" event={"ID":"1ace1cc9-f70e-4814-9542-8efb0b942784","Type":"ContainerStarted","Data":"4479924f229c0ead4efc28623a1cdef3ea385fb7f8f5e89af0dea31964108c03"}
Sep 17 22:14:30.103626 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:30.103603    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:14:30.125536 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:30.125491    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6" podStartSLOduration=1.125477074 podStartE2EDuration="1.125477074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:14:30.124312419 +0000 UTC m=+947.133241962" watchObservedRunningTime="2026-09-17 22:14:30.125477074 +0000 UTC m=+947.134406616"
Sep 17 22:14:50.642535 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.642497    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"]
Sep 17 22:14:50.647152 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.647125    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"
Sep 17 22:14:50.652132 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.652107    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk7tz\" (UniqueName: \"kubernetes.io/projected/99b47f73-ba61-4c3f-93c0-c39a88965295-kube-api-access-mk7tz\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq\" (UID: \"99b47f73-ba61-4c3f-93c0-c39a88965295\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"
Sep 17 22:14:50.652270 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.652147    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b47f73-ba61-4c3f-93c0-c39a88965295-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq\" (UID: \"99b47f73-ba61-4c3f-93c0-c39a88965295\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"
Sep 17 22:14:50.652500 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.652479    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-xv9nn\""
Sep 17 22:14:50.653552 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.653530    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 22:14:50.653683 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.653614    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-fd9d5\""
Sep 17 22:14:50.653683 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.653648    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 22:14:50.653784 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.653764    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 22:14:50.658912 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.658892    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"]
Sep 17 22:14:50.752904 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.752870    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b47f73-ba61-4c3f-93c0-c39a88965295-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq\" (UID: \"99b47f73-ba61-4c3f-93c0-c39a88965295\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"
Sep 17 22:14:50.753075 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.752952    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk7tz\" (UniqueName: \"kubernetes.io/projected/99b47f73-ba61-4c3f-93c0-c39a88965295-kube-api-access-mk7tz\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq\" (UID: \"99b47f73-ba61-4c3f-93c0-c39a88965295\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"
Sep 17 22:14:50.755257 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.755231    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b47f73-ba61-4c3f-93c0-c39a88965295-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq\" (UID: \"99b47f73-ba61-4c3f-93c0-c39a88965295\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"
Sep 17 22:14:50.760637 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.760615    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk7tz\" (UniqueName: \"kubernetes.io/projected/99b47f73-ba61-4c3f-93c0-c39a88965295-kube-api-access-mk7tz\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq\" (UID: \"99b47f73-ba61-4c3f-93c0-c39a88965295\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"
Sep 17 22:14:50.957159 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:50.957078    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"
Sep 17 22:14:51.226872 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:51.226849    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"]
Sep 17 22:14:51.229545 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:14:51.229515    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b47f73_ba61_4c3f_93c0_c39a88965295.slice/crio-43b5bb81940a710448a1bfdb69455e3c7fcb15547a4a54489c69ec5b674f466e WatchSource:0}: Error finding container 43b5bb81940a710448a1bfdb69455e3c7fcb15547a4a54489c69ec5b674f466e: Status 404 returned error can't find the container with id 43b5bb81940a710448a1bfdb69455e3c7fcb15547a4a54489c69ec5b674f466e
Sep 17 22:14:52.189102 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:52.189065    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq" event={"ID":"99b47f73-ba61-4c3f-93c0-c39a88965295","Type":"ContainerStarted","Data":"caa5269fc281a5bb48679bac7ccf58336c1747238d19e3e30f8597a8693d0b6a"}
Sep 17 22:14:52.189496 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:52.189125    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq" event={"ID":"99b47f73-ba61-4c3f-93c0-c39a88965295","Type":"ContainerStarted","Data":"43b5bb81940a710448a1bfdb69455e3c7fcb15547a4a54489c69ec5b674f466e"}
Sep 17 22:14:52.189496 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:52.189200    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"
Sep 17 22:14:52.204528 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:14:52.204486    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq" podStartSLOduration=2.204473843 podStartE2EDuration="2.204473843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:14:52.204125655 +0000 UTC m=+969.213055195" watchObservedRunningTime="2026-09-17 22:14:52.204473843 +0000 UTC m=+969.213403383"
Sep 17 22:15:01.113265 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:15:01.113227    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:15:23.195060 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:15:23.195034    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"
Sep 17 22:15:42.502777 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:15:42.502742    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"]
Sep 17 22:15:48.365849 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:15:48.365812    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"]
Sep 17 22:15:51.113571 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:15:51.113537    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6" podUID="1ace1cc9-f70e-4814-9542-8efb0b942784" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:15:53.194689 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:15:53.194652    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq" podUID="99b47f73-ba61-4c3f-93c0-c39a88965295" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:15:57.522022 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:15:57.521958    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6" podUID="1ace1cc9-f70e-4814-9542-8efb0b942784" containerName="main" containerID="cri-o://9a15c5a1eb0456ad452305ff9559ce4580468ff04b787f3b47f8d196ef4aae98" gracePeriod=45
Sep 17 22:16:01.113177 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:01.113130    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6" podUID="1ace1cc9-f70e-4814-9542-8efb0b942784" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:16:03.195115 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:03.195076    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq" podUID="99b47f73-ba61-4c3f-93c0-c39a88965295" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:16:03.385164 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:03.385098    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq" podUID="99b47f73-ba61-4c3f-93c0-c39a88965295" containerName="main" containerID="cri-o://caa5269fc281a5bb48679bac7ccf58336c1747238d19e3e30f8597a8693d0b6a" gracePeriod=45
Sep 17 22:16:11.113251 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:11.113202    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6" podUID="1ace1cc9-f70e-4814-9542-8efb0b942784" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:16:11.113651 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:11.113349    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:16:13.194902 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:13.194861    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq" podUID="99b47f73-ba61-4c3f-93c0-c39a88965295" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:16:13.195291 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:13.194959    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"
Sep 17 22:16:21.113174 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:21.113140    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6" podUID="1ace1cc9-f70e-4814-9542-8efb0b942784" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:16:23.195563 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:23.195526    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq" podUID="99b47f73-ba61-4c3f-93c0-c39a88965295" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:16:27.766820 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:27.766794    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:16:27.857194 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:27.857121    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace1cc9-f70e-4814-9542-8efb0b942784-tokenizer-cache\") pod \"1ace1cc9-f70e-4814-9542-8efb0b942784\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") "
Sep 17 22:16:27.857194 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:27.857167    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thdfj\" (UniqueName: \"kubernetes.io/projected/1ace1cc9-f70e-4814-9542-8efb0b942784-kube-api-access-thdfj\") pod \"1ace1cc9-f70e-4814-9542-8efb0b942784\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") "
Sep 17 22:16:27.857397 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:27.857245    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace1cc9-f70e-4814-9542-8efb0b942784-cachi2-cache\") pod \"1ace1cc9-f70e-4814-9542-8efb0b942784\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") "
Sep 17 22:16:27.857397 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:27.857271    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ace1cc9-f70e-4814-9542-8efb0b942784-tls-certs\") pod \"1ace1cc9-f70e-4814-9542-8efb0b942784\" (UID: \"1ace1cc9-f70e-4814-9542-8efb0b942784\") "
Sep 17 22:16:27.857488 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:27.857399    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ace1cc9-f70e-4814-9542-8efb0b942784-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "1ace1cc9-f70e-4814-9542-8efb0b942784" (UID: "1ace1cc9-f70e-4814-9542-8efb0b942784"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:16:27.857488 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:27.857463    2567 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace1cc9-f70e-4814-9542-8efb0b942784-tokenizer-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:16:27.857556 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:27.857513    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ace1cc9-f70e-4814-9542-8efb0b942784-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "1ace1cc9-f70e-4814-9542-8efb0b942784" (UID: "1ace1cc9-f70e-4814-9542-8efb0b942784"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:16:27.859291 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:27.859262    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ace1cc9-f70e-4814-9542-8efb0b942784-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ace1cc9-f70e-4814-9542-8efb0b942784" (UID: "1ace1cc9-f70e-4814-9542-8efb0b942784"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:16:27.859403 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:27.859310    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ace1cc9-f70e-4814-9542-8efb0b942784-kube-api-access-thdfj" (OuterVolumeSpecName: "kube-api-access-thdfj") pod "1ace1cc9-f70e-4814-9542-8efb0b942784" (UID: "1ace1cc9-f70e-4814-9542-8efb0b942784"). InnerVolumeSpecName "kube-api-access-thdfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:16:27.958422 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:27.958394    2567 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ace1cc9-f70e-4814-9542-8efb0b942784-cachi2-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:16:27.958422 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:27.958416    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ace1cc9-f70e-4814-9542-8efb0b942784-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:16:27.958422 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:27.958425    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thdfj\" (UniqueName: \"kubernetes.io/projected/1ace1cc9-f70e-4814-9542-8efb0b942784-kube-api-access-thdfj\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:16:28.550075 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:28.550035    2567 generic.go:356] "Generic (PLEG): container finished" podID="1ace1cc9-f70e-4814-9542-8efb0b942784" containerID="9a15c5a1eb0456ad452305ff9559ce4580468ff04b787f3b47f8d196ef4aae98" exitCode=0
Sep 17 22:16:28.550370 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:28.550083    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"
Sep 17 22:16:28.550370 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:28.550111    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6" event={"ID":"1ace1cc9-f70e-4814-9542-8efb0b942784","Type":"ContainerDied","Data":"9a15c5a1eb0456ad452305ff9559ce4580468ff04b787f3b47f8d196ef4aae98"}
Sep 17 22:16:28.550370 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:28.550155    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6" event={"ID":"1ace1cc9-f70e-4814-9542-8efb0b942784","Type":"ContainerDied","Data":"4479924f229c0ead4efc28623a1cdef3ea385fb7f8f5e89af0dea31964108c03"}
Sep 17 22:16:28.550370 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:28.550178    2567 scope.go:117] "RemoveContainer" containerID="9a15c5a1eb0456ad452305ff9559ce4580468ff04b787f3b47f8d196ef4aae98"
Sep 17 22:16:28.558241 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:28.558222    2567 scope.go:117] "RemoveContainer" containerID="9a15c5a1eb0456ad452305ff9559ce4580468ff04b787f3b47f8d196ef4aae98"
Sep 17 22:16:28.558470 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:16:28.558452    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a15c5a1eb0456ad452305ff9559ce4580468ff04b787f3b47f8d196ef4aae98\": container with ID starting with 9a15c5a1eb0456ad452305ff9559ce4580468ff04b787f3b47f8d196ef4aae98 not found: ID does not exist" containerID="9a15c5a1eb0456ad452305ff9559ce4580468ff04b787f3b47f8d196ef4aae98"
Sep 17 22:16:28.558519 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:28.558477    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a15c5a1eb0456ad452305ff9559ce4580468ff04b787f3b47f8d196ef4aae98"} err="failed to get container status \"9a15c5a1eb0456ad452305ff9559ce4580468ff04b787f3b47f8d196ef4aae98\": rpc error: code = NotFound desc = could not find container \"9a15c5a1eb0456ad452305ff9559ce4580468ff04b787f3b47f8d196ef4aae98\": container with ID starting with 9a15c5a1eb0456ad452305ff9559ce4580468ff04b787f3b47f8d196ef4aae98 not found: ID does not exist"
Sep 17 22:16:28.570373 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:28.570343    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"]
Sep 17 22:16:28.574770 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:28.574749    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffch9jb6"]
Sep 17 22:16:29.639368 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:29.639337    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ace1cc9-f70e-4814-9542-8efb0b942784" path="/var/lib/kubelet/pods/1ace1cc9-f70e-4814-9542-8efb0b942784/volumes"
Sep 17 22:16:33.194992 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:33.194953    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq" podUID="99b47f73-ba61-4c3f-93c0-c39a88965295" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:16:33.571551 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:33.571516    2567 generic.go:356] "Generic (PLEG): container finished" podID="99b47f73-ba61-4c3f-93c0-c39a88965295" containerID="caa5269fc281a5bb48679bac7ccf58336c1747238d19e3e30f8597a8693d0b6a" exitCode=0
Sep 17 22:16:33.571706 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:33.571582    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq" event={"ID":"99b47f73-ba61-4c3f-93c0-c39a88965295","Type":"ContainerDied","Data":"caa5269fc281a5bb48679bac7ccf58336c1747238d19e3e30f8597a8693d0b6a"}
Sep 17 22:16:33.623043 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:33.623019    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"
Sep 17 22:16:33.699230 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:33.699187    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk7tz\" (UniqueName: \"kubernetes.io/projected/99b47f73-ba61-4c3f-93c0-c39a88965295-kube-api-access-mk7tz\") pod \"99b47f73-ba61-4c3f-93c0-c39a88965295\" (UID: \"99b47f73-ba61-4c3f-93c0-c39a88965295\") "
Sep 17 22:16:33.699230 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:33.699238    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b47f73-ba61-4c3f-93c0-c39a88965295-tls-certs\") pod \"99b47f73-ba61-4c3f-93c0-c39a88965295\" (UID: \"99b47f73-ba61-4c3f-93c0-c39a88965295\") "
Sep 17 22:16:33.701394 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:33.701361    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b47f73-ba61-4c3f-93c0-c39a88965295-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99b47f73-ba61-4c3f-93c0-c39a88965295" (UID: "99b47f73-ba61-4c3f-93c0-c39a88965295"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:16:33.701394 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:33.701386    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b47f73-ba61-4c3f-93c0-c39a88965295-kube-api-access-mk7tz" (OuterVolumeSpecName: "kube-api-access-mk7tz") pod "99b47f73-ba61-4c3f-93c0-c39a88965295" (UID: "99b47f73-ba61-4c3f-93c0-c39a88965295"). InnerVolumeSpecName "kube-api-access-mk7tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:16:33.799946 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:33.799861    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk7tz\" (UniqueName: \"kubernetes.io/projected/99b47f73-ba61-4c3f-93c0-c39a88965295-kube-api-access-mk7tz\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:16:33.799946 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:33.799897    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b47f73-ba61-4c3f-93c0-c39a88965295-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:16:34.575987 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:34.575953    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"
Sep 17 22:16:34.575987 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:34.575968    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq" event={"ID":"99b47f73-ba61-4c3f-93c0-c39a88965295","Type":"ContainerDied","Data":"43b5bb81940a710448a1bfdb69455e3c7fcb15547a4a54489c69ec5b674f466e"}
Sep 17 22:16:34.576479 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:34.576036    2567 scope.go:117] "RemoveContainer" containerID="caa5269fc281a5bb48679bac7ccf58336c1747238d19e3e30f8597a8693d0b6a"
Sep 17 22:16:34.597423 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:34.597402    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"]
Sep 17 22:16:34.602582 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:34.602564    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-vjmkq"]
Sep 17 22:16:35.639191 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:35.639160    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99b47f73-ba61-4c3f-93c0-c39a88965295" path="/var/lib/kubelet/pods/99b47f73-ba61-4c3f-93c0-c39a88965295/volumes"
Sep 17 22:16:48.466965 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.466934    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"]
Sep 17 22:16:48.467436 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.467230    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ace1cc9-f70e-4814-9542-8efb0b942784" containerName="main"
Sep 17 22:16:48.467436 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.467243    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ace1cc9-f70e-4814-9542-8efb0b942784" containerName="main"
Sep 17 22:16:48.467436 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.467252    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b47f73-ba61-4c3f-93c0-c39a88965295" containerName="main"
Sep 17 22:16:48.467436 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.467257    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b47f73-ba61-4c3f-93c0-c39a88965295" containerName="main"
Sep 17 22:16:48.467436 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.467300    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ace1cc9-f70e-4814-9542-8efb0b942784" containerName="main"
Sep 17 22:16:48.467436 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.467306    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b47f73-ba61-4c3f-93c0-c39a88965295" containerName="main"
Sep 17 22:16:48.471320 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.471300    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.474817 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.474784    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 22:16:48.474943 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.474822    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 22:16:48.475026 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.474949    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 22:16:48.475084 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.474971    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-fkrzb\""
Sep 17 22:16:48.475180 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.475159    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-gs6zp\""
Sep 17 22:16:48.483899 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.483876    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"]
Sep 17 22:16:48.501837 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.501818    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.501945 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.501849    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct865\" (UniqueName: \"kubernetes.io/projected/b77e5431-e7d4-4844-963a-c18037e0f130-kube-api-access-ct865\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.501945 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.501880    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.502067 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.501955    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b77e5431-e7d4-4844-963a-c18037e0f130-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.502067 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.502028    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.502179 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.502066    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.603164 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.603136    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b77e5431-e7d4-4844-963a-c18037e0f130-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.603264 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.603175    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.603264 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.603203    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.603264 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.603230    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.603264 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.603248    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct865\" (UniqueName: \"kubernetes.io/projected/b77e5431-e7d4-4844-963a-c18037e0f130-kube-api-access-ct865\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.603461 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.603266    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.603523 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.603483    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.603751 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.603721    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.603831 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.603770    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.605393 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.605367    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.605678 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.605660    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b77e5431-e7d4-4844-963a-c18037e0f130-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.616500 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.616480    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct865\" (UniqueName: \"kubernetes.io/projected/b77e5431-e7d4-4844-963a-c18037e0f130-kube-api-access-ct865\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.780822 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.780799    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:48.785221 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.785198    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"]
Sep 17 22:16:48.788257 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.788240    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"
Sep 17 22:16:48.790864 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.790845    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-khqrr\""
Sep 17 22:16:48.799335 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.799280    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"]
Sep 17 22:16:48.904675 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.904643    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdjsm\" (UniqueName: \"kubernetes.io/projected/4c3b149c-6b9c-4a8a-bc90-104db4643ade-kube-api-access-kdjsm\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59\" (UID: \"4c3b149c-6b9c-4a8a-bc90-104db4643ade\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"
Sep 17 22:16:48.904675 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:48.904679    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c3b149c-6b9c-4a8a-bc90-104db4643ade-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59\" (UID: \"4c3b149c-6b9c-4a8a-bc90-104db4643ade\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"
Sep 17 22:16:49.005866 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:49.005834    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdjsm\" (UniqueName: \"kubernetes.io/projected/4c3b149c-6b9c-4a8a-bc90-104db4643ade-kube-api-access-kdjsm\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59\" (UID: \"4c3b149c-6b9c-4a8a-bc90-104db4643ade\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"
Sep 17 22:16:49.006019 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:49.005871    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c3b149c-6b9c-4a8a-bc90-104db4643ade-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59\" (UID: \"4c3b149c-6b9c-4a8a-bc90-104db4643ade\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"
Sep 17 22:16:49.008276 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:49.008257    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c3b149c-6b9c-4a8a-bc90-104db4643ade-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59\" (UID: \"4c3b149c-6b9c-4a8a-bc90-104db4643ade\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"
Sep 17 22:16:49.013536 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:49.013520    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdjsm\" (UniqueName: \"kubernetes.io/projected/4c3b149c-6b9c-4a8a-bc90-104db4643ade-kube-api-access-kdjsm\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59\" (UID: \"4c3b149c-6b9c-4a8a-bc90-104db4643ade\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"
Sep 17 22:16:49.051856 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:49.051835    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"]
Sep 17 22:16:49.053860 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:16:49.053835    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77e5431_e7d4_4844_963a_c18037e0f130.slice/crio-54d01c47be4df5a02e884da27f0484432f1629d1d9e3c5ad9f0ac45f5c60561a WatchSource:0}: Error finding container 54d01c47be4df5a02e884da27f0484432f1629d1d9e3c5ad9f0ac45f5c60561a: Status 404 returned error can't find the container with id 54d01c47be4df5a02e884da27f0484432f1629d1d9e3c5ad9f0ac45f5c60561a
Sep 17 22:16:49.112373 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:49.112347    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"
Sep 17 22:16:49.385402 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:49.379970    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"]
Sep 17 22:16:49.629320 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:49.629243    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59" event={"ID":"4c3b149c-6b9c-4a8a-bc90-104db4643ade","Type":"ContainerStarted","Data":"b277834ba135ff1a9b1e83069d9d53c169b3391a5347cdc71df04e6ed3ccd110"}
Sep 17 22:16:49.629320 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:49.629299    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59" event={"ID":"4c3b149c-6b9c-4a8a-bc90-104db4643ade","Type":"ContainerStarted","Data":"1f528fc40c8e9f7ac96867f93f120d45267c87f83ebd6c51e1775ba6c1f5b46c"}
Sep 17 22:16:49.629756 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:49.629346    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"
Sep 17 22:16:49.630424 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:49.630399    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl" event={"ID":"b77e5431-e7d4-4844-963a-c18037e0f130","Type":"ContainerStarted","Data":"54d01c47be4df5a02e884da27f0484432f1629d1d9e3c5ad9f0ac45f5c60561a"}
Sep 17 22:16:49.646660 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:49.646614    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59" podStartSLOduration=1.646602739 podStartE2EDuration="1.646602739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:16:49.645063277 +0000 UTC m=+1086.653992821" watchObservedRunningTime="2026-09-17 22:16:49.646602739 +0000 UTC m=+1086.655532279"
Sep 17 22:16:50.635413 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:50.635374    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl" event={"ID":"b77e5431-e7d4-4844-963a-c18037e0f130","Type":"ContainerStarted","Data":"7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217"}
Sep 17 22:16:51.641495 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:51.641460    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl" event={"ID":"b77e5431-e7d4-4844-963a-c18037e0f130","Type":"ContainerStarted","Data":"5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018"}
Sep 17 22:16:51.641949 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:51.641565    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:51.661674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:51.661571    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl" podStartSLOduration=2.277209289 podStartE2EDuration="3.66155246s" totalImagesPullingTime="1.384343171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:16:51.658890901 +0000 UTC m=+1088.667820441" watchObservedRunningTime="2026-09-17 22:16:51.66155246 +0000 UTC m=+1088.670482003"
Sep 17 22:16:51.781745 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:51.781720    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:58.781605 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:58.781568    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:58.793686 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:58.793664    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:16:59.678263 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:16:59.678226    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:17:00.682836 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:00.682807    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:17:20.637522 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:20.637493    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"
Sep 17 22:17:43.748025 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:43.747976    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"]
Sep 17 22:17:43.754640 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:43.754608    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"]
Sep 17 22:17:43.758947 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:17:43.758913    2567 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:17:43.758947 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:17:43.758947 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:17:43.758947 ip-10-0-142-197 kubenswrapper[2567]:  > containerID="5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018" cmd=["/bin/sleep","15"]
Sep 17 22:17:43.759223 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:43.758969    2567 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl" message="" err=<
Sep 17 22:17:43.759223 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:17:43.759223 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:17:43.759223 ip-10-0-142-197 kubenswrapper[2567]:  >
Sep 17 22:17:43.759223 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:17:43.758986    2567 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:17:43.759223 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:17:43.759223 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:17:43.759223 ip-10-0-142-197 kubenswrapper[2567]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl" podUID="b77e5431-e7d4-4844-963a-c18037e0f130" containerName="main" containerID="cri-o://5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018"
Sep 17 22:17:43.759223 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:43.759047    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl" podUID="b77e5431-e7d4-4844-963a-c18037e0f130" containerName="main" containerID="cri-o://5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018" gracePeriod=60
Sep 17 22:17:43.973859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:43.973818    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl" podUID="b77e5431-e7d4-4844-963a-c18037e0f130" containerName="llm-d-routing-sidecar" containerID="cri-o://7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217" gracePeriod=60
Sep 17 22:17:44.208941 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.208917    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:17:44.309673 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.309648    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-tmp-dir\") pod \"b77e5431-e7d4-4844-963a-c18037e0f130\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") "
Sep 17 22:17:44.309820 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.309700    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-model-cache\") pod \"b77e5431-e7d4-4844-963a-c18037e0f130\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") "
Sep 17 22:17:44.309820 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.309729    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b77e5431-e7d4-4844-963a-c18037e0f130-tls-certs\") pod \"b77e5431-e7d4-4844-963a-c18037e0f130\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") "
Sep 17 22:17:44.309820 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.309746    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-dshm\") pod \"b77e5431-e7d4-4844-963a-c18037e0f130\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") "
Sep 17 22:17:44.309820 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.309790    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-home\") pod \"b77e5431-e7d4-4844-963a-c18037e0f130\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") "
Sep 17 22:17:44.309820 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.309809    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct865\" (UniqueName: \"kubernetes.io/projected/b77e5431-e7d4-4844-963a-c18037e0f130-kube-api-access-ct865\") pod \"b77e5431-e7d4-4844-963a-c18037e0f130\" (UID: \"b77e5431-e7d4-4844-963a-c18037e0f130\") "
Sep 17 22:17:44.310137 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.309913    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b77e5431-e7d4-4844-963a-c18037e0f130" (UID: "b77e5431-e7d4-4844-963a-c18037e0f130"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:44.310137 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.310036    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-home" (OuterVolumeSpecName: "home") pod "b77e5431-e7d4-4844-963a-c18037e0f130" (UID: "b77e5431-e7d4-4844-963a-c18037e0f130"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:44.310137 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.310040    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:17:44.310137 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.310055    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-model-cache" (OuterVolumeSpecName: "model-cache") pod "b77e5431-e7d4-4844-963a-c18037e0f130" (UID: "b77e5431-e7d4-4844-963a-c18037e0f130"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:44.311955 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.311935    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b77e5431-e7d4-4844-963a-c18037e0f130-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b77e5431-e7d4-4844-963a-c18037e0f130" (UID: "b77e5431-e7d4-4844-963a-c18037e0f130"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:17:44.312068 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.311956    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-dshm" (OuterVolumeSpecName: "dshm") pod "b77e5431-e7d4-4844-963a-c18037e0f130" (UID: "b77e5431-e7d4-4844-963a-c18037e0f130"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:17:44.312068 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.311975    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77e5431-e7d4-4844-963a-c18037e0f130-kube-api-access-ct865" (OuterVolumeSpecName: "kube-api-access-ct865") pod "b77e5431-e7d4-4844-963a-c18037e0f130" (UID: "b77e5431-e7d4-4844-963a-c18037e0f130"). InnerVolumeSpecName "kube-api-access-ct865". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:17:44.410380 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.410327    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct865\" (UniqueName: \"kubernetes.io/projected/b77e5431-e7d4-4844-963a-c18037e0f130-kube-api-access-ct865\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:17:44.410380 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.410350    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:17:44.410380 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.410361    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b77e5431-e7d4-4844-963a-c18037e0f130-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:17:44.410380 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.410368    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:17:44.410380 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.410375    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b77e5431-e7d4-4844-963a-c18037e0f130-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:17:44.828592 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.828551    2567 generic.go:356] "Generic (PLEG): container finished" podID="b77e5431-e7d4-4844-963a-c18037e0f130" containerID="5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018" exitCode=0
Sep 17 22:17:44.828592 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.828588    2567 generic.go:356] "Generic (PLEG): container finished" podID="b77e5431-e7d4-4844-963a-c18037e0f130" containerID="7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217" exitCode=0
Sep 17 22:17:44.829130 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.828602    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"
Sep 17 22:17:44.829130 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.828635    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl" event={"ID":"b77e5431-e7d4-4844-963a-c18037e0f130","Type":"ContainerDied","Data":"5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018"}
Sep 17 22:17:44.829130 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.828702    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl" event={"ID":"b77e5431-e7d4-4844-963a-c18037e0f130","Type":"ContainerDied","Data":"7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217"}
Sep 17 22:17:44.829130 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.828735    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl" event={"ID":"b77e5431-e7d4-4844-963a-c18037e0f130","Type":"ContainerDied","Data":"54d01c47be4df5a02e884da27f0484432f1629d1d9e3c5ad9f0ac45f5c60561a"}
Sep 17 22:17:44.829130 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.828764    2567 scope.go:117] "RemoveContainer" containerID="5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018"
Sep 17 22:17:44.836770 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.836755    2567 scope.go:117] "RemoveContainer" containerID="7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217"
Sep 17 22:17:44.843692 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.843676    2567 scope.go:117] "RemoveContainer" containerID="5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018"
Sep 17 22:17:44.843926 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:17:44.843909    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018\": container with ID starting with 5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018 not found: ID does not exist" containerID="5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018"
Sep 17 22:17:44.843984 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.843933    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018"} err="failed to get container status \"5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018\": rpc error: code = NotFound desc = could not find container \"5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018\": container with ID starting with 5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018 not found: ID does not exist"
Sep 17 22:17:44.843984 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.843949    2567 scope.go:117] "RemoveContainer" containerID="7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217"
Sep 17 22:17:44.844196 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:17:44.844180    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217\": container with ID starting with 7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217 not found: ID does not exist" containerID="7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217"
Sep 17 22:17:44.844241 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.844200    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217"} err="failed to get container status \"7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217\": rpc error: code = NotFound desc = could not find container \"7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217\": container with ID starting with 7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217 not found: ID does not exist"
Sep 17 22:17:44.844241 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.844215    2567 scope.go:117] "RemoveContainer" containerID="5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018"
Sep 17 22:17:44.844414 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.844400    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018"} err="failed to get container status \"5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018\": rpc error: code = NotFound desc = could not find container \"5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018\": container with ID starting with 5c99ff64113819fb954d35a40dd6009a9f02081da442be6bd42d6517ddccf018 not found: ID does not exist"
Sep 17 22:17:44.844503 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.844413    2567 scope.go:117] "RemoveContainer" containerID="7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217"
Sep 17 22:17:44.844650 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.844630    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217"} err="failed to get container status \"7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217\": rpc error: code = NotFound desc = could not find container \"7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217\": container with ID starting with 7d1c5058037b1e75055db3904ed1db260eeccaf5927cc4d409ba73b8b3800217 not found: ID does not exist"
Sep 17 22:17:44.847747 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.847726    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"]
Sep 17 22:17:44.851693 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:44.851672    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-6q2rl"]
Sep 17 22:17:45.639390 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:45.639361    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b77e5431-e7d4-4844-963a-c18037e0f130" path="/var/lib/kubelet/pods/b77e5431-e7d4-4844-963a-c18037e0f130/volumes"
Sep 17 22:17:50.637156 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:50.637124    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59" podUID="4c3b149c-6b9c-4a8a-bc90-104db4643ade" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:17:58.773744 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:17:58.773691    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59" podUID="4c3b149c-6b9c-4a8a-bc90-104db4643ade" containerName="main" containerID="cri-o://b277834ba135ff1a9b1e83069d9d53c169b3391a5347cdc71df04e6ed3ccd110" gracePeriod=45
Sep 17 22:18:00.637567 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:00.637521    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59" podUID="4c3b149c-6b9c-4a8a-bc90-104db4643ade" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:18:06.619617 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.619533    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"]
Sep 17 22:18:06.620054 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.619793    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b77e5431-e7d4-4844-963a-c18037e0f130" containerName="llm-d-routing-sidecar"
Sep 17 22:18:06.620054 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.619804    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b77e5431-e7d4-4844-963a-c18037e0f130" containerName="llm-d-routing-sidecar"
Sep 17 22:18:06.620054 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.619814    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b77e5431-e7d4-4844-963a-c18037e0f130" containerName="main"
Sep 17 22:18:06.620054 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.619820    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b77e5431-e7d4-4844-963a-c18037e0f130" containerName="main"
Sep 17 22:18:06.620054 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.619863    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b77e5431-e7d4-4844-963a-c18037e0f130" containerName="llm-d-routing-sidecar"
Sep 17 22:18:06.620054 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.619870    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b77e5431-e7d4-4844-963a-c18037e0f130" containerName="main"
Sep 17 22:18:06.624212 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.624192    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"
Sep 17 22:18:06.626908 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.626887    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 22:18:06.627994 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.627965    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 22:18:06.627994 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.627983    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-t4dgq\""
Sep 17 22:18:06.628174 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.627983    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 22:18:06.628174 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.627976    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-wwc9t\""
Sep 17 22:18:06.635222 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.635202    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"]
Sep 17 22:18:06.667973 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.667947    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz2k2\" (UniqueName: \"kubernetes.io/projected/3ce68e03-8697-4388-8c36-f3d3f59ad634-kube-api-access-jz2k2\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc\" (UID: \"3ce68e03-8697-4388-8c36-f3d3f59ad634\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"
Sep 17 22:18:06.668083 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.667991    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce68e03-8697-4388-8c36-f3d3f59ad634-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc\" (UID: \"3ce68e03-8697-4388-8c36-f3d3f59ad634\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"
Sep 17 22:18:06.768798 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.768777    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz2k2\" (UniqueName: \"kubernetes.io/projected/3ce68e03-8697-4388-8c36-f3d3f59ad634-kube-api-access-jz2k2\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc\" (UID: \"3ce68e03-8697-4388-8c36-f3d3f59ad634\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"
Sep 17 22:18:06.768879 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.768812    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce68e03-8697-4388-8c36-f3d3f59ad634-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc\" (UID: \"3ce68e03-8697-4388-8c36-f3d3f59ad634\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"
Sep 17 22:18:06.771172 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.771155    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce68e03-8697-4388-8c36-f3d3f59ad634-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc\" (UID: \"3ce68e03-8697-4388-8c36-f3d3f59ad634\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"
Sep 17 22:18:06.776100 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.776080    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz2k2\" (UniqueName: \"kubernetes.io/projected/3ce68e03-8697-4388-8c36-f3d3f59ad634-kube-api-access-jz2k2\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc\" (UID: \"3ce68e03-8697-4388-8c36-f3d3f59ad634\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"
Sep 17 22:18:06.933636 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:06.933583    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"
Sep 17 22:18:07.201951 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:07.201927    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"]
Sep 17 22:18:07.204511 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:18:07.204489    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce68e03_8697_4388_8c36_f3d3f59ad634.slice/crio-18c8895b5236e2cd134c6055343bbda70d242c699b2410513172ef8c743781d5 WatchSource:0}: Error finding container 18c8895b5236e2cd134c6055343bbda70d242c699b2410513172ef8c743781d5: Status 404 returned error can't find the container with id 18c8895b5236e2cd134c6055343bbda70d242c699b2410513172ef8c743781d5
Sep 17 22:18:07.914074 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:07.914034    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc" event={"ID":"3ce68e03-8697-4388-8c36-f3d3f59ad634","Type":"ContainerStarted","Data":"97affdd481102b7d3ce9cc5ba8173f7bed6cae85d723c61ca9bd6360a2c5f234"}
Sep 17 22:18:07.914468 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:07.914124    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc" event={"ID":"3ce68e03-8697-4388-8c36-f3d3f59ad634","Type":"ContainerStarted","Data":"18c8895b5236e2cd134c6055343bbda70d242c699b2410513172ef8c743781d5"}
Sep 17 22:18:07.914468 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:07.914149    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"
Sep 17 22:18:07.930210 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:07.930163    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc" podStartSLOduration=1.930132664 podStartE2EDuration="1.930132664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:18:07.928628495 +0000 UTC m=+1164.937558035" watchObservedRunningTime="2026-09-17 22:18:07.930132664 +0000 UTC m=+1164.939062206"
Sep 17 22:18:10.637528 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:10.637496    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59" podUID="4c3b149c-6b9c-4a8a-bc90-104db4643ade" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:18:10.637889 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:10.637592    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"
Sep 17 22:18:20.636703 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:20.636663    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59" podUID="4c3b149c-6b9c-4a8a-bc90-104db4643ade" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:18:28.997881 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:28.997850    2567 generic.go:356] "Generic (PLEG): container finished" podID="4c3b149c-6b9c-4a8a-bc90-104db4643ade" containerID="b277834ba135ff1a9b1e83069d9d53c169b3391a5347cdc71df04e6ed3ccd110" exitCode=0
Sep 17 22:18:28.998194 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:28.997881    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59" event={"ID":"4c3b149c-6b9c-4a8a-bc90-104db4643ade","Type":"ContainerDied","Data":"b277834ba135ff1a9b1e83069d9d53c169b3391a5347cdc71df04e6ed3ccd110"}
Sep 17 22:18:28.998194 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:28.997973    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59" event={"ID":"4c3b149c-6b9c-4a8a-bc90-104db4643ade","Type":"ContainerDied","Data":"1f528fc40c8e9f7ac96867f93f120d45267c87f83ebd6c51e1775ba6c1f5b46c"}
Sep 17 22:18:28.998194 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:28.997989    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f528fc40c8e9f7ac96867f93f120d45267c87f83ebd6c51e1775ba6c1f5b46c"
Sep 17 22:18:29.001923 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:29.001905    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"
Sep 17 22:18:29.120406 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:29.120351    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c3b149c-6b9c-4a8a-bc90-104db4643ade-tls-certs\") pod \"4c3b149c-6b9c-4a8a-bc90-104db4643ade\" (UID: \"4c3b149c-6b9c-4a8a-bc90-104db4643ade\") "
Sep 17 22:18:29.120533 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:29.120411    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdjsm\" (UniqueName: \"kubernetes.io/projected/4c3b149c-6b9c-4a8a-bc90-104db4643ade-kube-api-access-kdjsm\") pod \"4c3b149c-6b9c-4a8a-bc90-104db4643ade\" (UID: \"4c3b149c-6b9c-4a8a-bc90-104db4643ade\") "
Sep 17 22:18:29.122328 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:29.122304    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c3b149c-6b9c-4a8a-bc90-104db4643ade-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c3b149c-6b9c-4a8a-bc90-104db4643ade" (UID: "4c3b149c-6b9c-4a8a-bc90-104db4643ade"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:18:29.122398 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:29.122369    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3b149c-6b9c-4a8a-bc90-104db4643ade-kube-api-access-kdjsm" (OuterVolumeSpecName: "kube-api-access-kdjsm") pod "4c3b149c-6b9c-4a8a-bc90-104db4643ade" (UID: "4c3b149c-6b9c-4a8a-bc90-104db4643ade"). InnerVolumeSpecName "kube-api-access-kdjsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:18:29.221462 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:29.221440    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c3b149c-6b9c-4a8a-bc90-104db4643ade-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:18:29.221462 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:29.221462    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdjsm\" (UniqueName: \"kubernetes.io/projected/4c3b149c-6b9c-4a8a-bc90-104db4643ade-kube-api-access-kdjsm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:18:30.001493 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:30.001465    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"
Sep 17 22:18:30.016410 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:30.016379    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"]
Sep 17 22:18:30.019795 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:30.019775    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75bch59"]
Sep 17 22:18:31.639247 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:31.639218    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c3b149c-6b9c-4a8a-bc90-104db4643ade" path="/var/lib/kubelet/pods/4c3b149c-6b9c-4a8a-bc90-104db4643ade/volumes"
Sep 17 22:18:38.920345 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:38.920317    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"
Sep 17 22:18:44.477219 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.477188    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"]
Sep 17 22:18:44.477701 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.477464    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c3b149c-6b9c-4a8a-bc90-104db4643ade" containerName="main"
Sep 17 22:18:44.477701 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.477477    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c3b149c-6b9c-4a8a-bc90-104db4643ade" containerName="main"
Sep 17 22:18:44.477701 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.477531    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c3b149c-6b9c-4a8a-bc90-104db4643ade" containerName="main"
Sep 17 22:18:44.481587 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.481568    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.485938 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.485544    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-xdl4w\""
Sep 17 22:18:44.485938 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.485603    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-d5jld\""
Sep 17 22:18:44.485938 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.485608    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 22:18:44.485938 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.485608    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 22:18:44.485938 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.485777    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 22:18:44.501125 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.501102    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"]
Sep 17 22:18:44.526983 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.526953    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.527109 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.527035    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.527109 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.527066    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.527109 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.527103    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.527221 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.527142    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54g4z\" (UniqueName: \"kubernetes.io/projected/58be1ae1-a5e1-490c-a0ae-4602a8c22092-kube-api-access-54g4z\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.527221 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.527170    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58be1ae1-a5e1-490c-a0ae-4602a8c22092-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.627814 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.627782    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.627968 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.627827    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.627968 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.627849    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.627968 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.627883    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.627968 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.627917    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54g4z\" (UniqueName: \"kubernetes.io/projected/58be1ae1-a5e1-490c-a0ae-4602a8c22092-kube-api-access-54g4z\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.627968 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.627944    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58be1ae1-a5e1-490c-a0ae-4602a8c22092-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.628278 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.628255    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.628340 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.628286    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.628340 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.628305    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.630300 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.630268    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.630473 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.630455    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58be1ae1-a5e1-490c-a0ae-4602a8c22092-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.635549 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.635532    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54g4z\" (UniqueName: \"kubernetes.io/projected/58be1ae1-a5e1-490c-a0ae-4602a8c22092-kube-api-access-54g4z\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:44.793517 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:44.793492    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:45.067621 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:45.067595    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"]
Sep 17 22:18:45.069555 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:18:45.069531    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58be1ae1_a5e1_490c_a0ae_4602a8c22092.slice/crio-58e4aa97fc2df7790768dff944167896df96adf79dab4b4a09cea415a2a45003 WatchSource:0}: Error finding container 58e4aa97fc2df7790768dff944167896df96adf79dab4b4a09cea415a2a45003: Status 404 returned error can't find the container with id 58e4aa97fc2df7790768dff944167896df96adf79dab4b4a09cea415a2a45003
Sep 17 22:18:46.061126 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:46.061085    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8" event={"ID":"58be1ae1-a5e1-490c-a0ae-4602a8c22092","Type":"ContainerStarted","Data":"55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0"}
Sep 17 22:18:46.061543 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:46.061154    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8" event={"ID":"58be1ae1-a5e1-490c-a0ae-4602a8c22092","Type":"ContainerStarted","Data":"58e4aa97fc2df7790768dff944167896df96adf79dab4b4a09cea415a2a45003"}
Sep 17 22:18:46.061543 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:46.061193    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:47.066417 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:47.066386    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8" event={"ID":"58be1ae1-a5e1-490c-a0ae-4602a8c22092","Type":"ContainerStarted","Data":"0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754"}
Sep 17 22:18:47.087426 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:47.087380    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8" podStartSLOduration=3.087367373 podStartE2EDuration="3.087367373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:18:47.085616601 +0000 UTC m=+1204.094546143" watchObservedRunningTime="2026-09-17 22:18:47.087367373 +0000 UTC m=+1204.096296913"
Sep 17 22:18:47.793708 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:47.793679    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:54.794212 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:54.794177    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:54.806814 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:54.806785    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:55.106325 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:55.106246    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:18:55.107519 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:18:55.107500    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:19:21.364441 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:21.364409    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"]
Sep 17 22:19:21.373149 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:19:21.373113    2567 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:19:21.373149 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:19:21.373149 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:19:21.373149 ip-10-0-142-197 kubenswrapper[2567]:  > containerID="0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754" cmd=["/bin/sleep","15"]
Sep 17 22:19:21.373414 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:21.373158    2567 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8" message="" err=<
Sep 17 22:19:21.373414 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:19:21.373414 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:19:21.373414 ip-10-0-142-197 kubenswrapper[2567]:  >
Sep 17 22:19:21.373414 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:19:21.373169    2567 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:19:21.373414 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:19:21.373414 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:19:21.373414 ip-10-0-142-197 kubenswrapper[2567]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8" podUID="58be1ae1-a5e1-490c-a0ae-4602a8c22092" containerName="main" containerID="cri-o://0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754"
Sep 17 22:19:21.373414 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:21.373199    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8" podUID="58be1ae1-a5e1-490c-a0ae-4602a8c22092" containerName="main" containerID="cri-o://0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754" gracePeriod=60
Sep 17 22:19:21.589026 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:21.588980    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8" podUID="58be1ae1-a5e1-490c-a0ae-4602a8c22092" containerName="llm-d-routing-sidecar" containerID="cri-o://55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0" gracePeriod=60
Sep 17 22:19:21.825824 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:21.825801    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:19:21.999314 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:21.999295    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-dshm\") pod \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") "
Sep 17 22:19:21.999474 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:21.999328    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-tmp-dir\") pod \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") "
Sep 17 22:19:21.999474 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:21.999379    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-home\") pod \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") "
Sep 17 22:19:21.999474 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:21.999404    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-model-cache\") pod \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") "
Sep 17 22:19:21.999474 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:21.999436    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54g4z\" (UniqueName: \"kubernetes.io/projected/58be1ae1-a5e1-490c-a0ae-4602a8c22092-kube-api-access-54g4z\") pod \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") "
Sep 17 22:19:21.999474 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:21.999451    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58be1ae1-a5e1-490c-a0ae-4602a8c22092-tls-certs\") pod \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\" (UID: \"58be1ae1-a5e1-490c-a0ae-4602a8c22092\") "
Sep 17 22:19:21.999737 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:21.999599    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58be1ae1-a5e1-490c-a0ae-4602a8c22092" (UID: "58be1ae1-a5e1-490c-a0ae-4602a8c22092"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:21.999737 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:21.999645    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-home" (OuterVolumeSpecName: "home") pod "58be1ae1-a5e1-490c-a0ae-4602a8c22092" (UID: "58be1ae1-a5e1-490c-a0ae-4602a8c22092"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:21.999737 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:21.999711    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-model-cache" (OuterVolumeSpecName: "model-cache") pod "58be1ae1-a5e1-490c-a0ae-4602a8c22092" (UID: "58be1ae1-a5e1-490c-a0ae-4602a8c22092"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:22.001535 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.001509    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58be1ae1-a5e1-490c-a0ae-4602a8c22092-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58be1ae1-a5e1-490c-a0ae-4602a8c22092" (UID: "58be1ae1-a5e1-490c-a0ae-4602a8c22092"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:19:22.001618 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.001535    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-dshm" (OuterVolumeSpecName: "dshm") pod "58be1ae1-a5e1-490c-a0ae-4602a8c22092" (UID: "58be1ae1-a5e1-490c-a0ae-4602a8c22092"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:22.001618 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.001564    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58be1ae1-a5e1-490c-a0ae-4602a8c22092-kube-api-access-54g4z" (OuterVolumeSpecName: "kube-api-access-54g4z") pod "58be1ae1-a5e1-490c-a0ae-4602a8c22092" (UID: "58be1ae1-a5e1-490c-a0ae-4602a8c22092"). InnerVolumeSpecName "kube-api-access-54g4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:19:22.100084 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.100064    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54g4z\" (UniqueName: \"kubernetes.io/projected/58be1ae1-a5e1-490c-a0ae-4602a8c22092-kube-api-access-54g4z\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:19:22.100084 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.100085    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58be1ae1-a5e1-490c-a0ae-4602a8c22092-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:19:22.100197 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.100095    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:19:22.100197 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.100124    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:19:22.100197 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.100133    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:19:22.100197 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.100140    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be1ae1-a5e1-490c-a0ae-4602a8c22092-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:19:22.202658 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.202636    2567 generic.go:356] "Generic (PLEG): container finished" podID="58be1ae1-a5e1-490c-a0ae-4602a8c22092" containerID="0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754" exitCode=0
Sep 17 22:19:22.202658 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.202654    2567 generic.go:356] "Generic (PLEG): container finished" podID="58be1ae1-a5e1-490c-a0ae-4602a8c22092" containerID="55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0" exitCode=0
Sep 17 22:19:22.202777 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.202678    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"
Sep 17 22:19:22.202777 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.202718    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8" event={"ID":"58be1ae1-a5e1-490c-a0ae-4602a8c22092","Type":"ContainerDied","Data":"0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754"}
Sep 17 22:19:22.202777 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.202773    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8" event={"ID":"58be1ae1-a5e1-490c-a0ae-4602a8c22092","Type":"ContainerDied","Data":"55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0"}
Sep 17 22:19:22.202863 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.202792    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8" event={"ID":"58be1ae1-a5e1-490c-a0ae-4602a8c22092","Type":"ContainerDied","Data":"58e4aa97fc2df7790768dff944167896df96adf79dab4b4a09cea415a2a45003"}
Sep 17 22:19:22.202863 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.202813    2567 scope.go:117] "RemoveContainer" containerID="0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754"
Sep 17 22:19:22.210589 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.210571    2567 scope.go:117] "RemoveContainer" containerID="55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0"
Sep 17 22:19:22.217199 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.217182    2567 scope.go:117] "RemoveContainer" containerID="0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754"
Sep 17 22:19:22.217455 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:19:22.217433    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754\": container with ID starting with 0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754 not found: ID does not exist" containerID="0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754"
Sep 17 22:19:22.217507 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.217461    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754"} err="failed to get container status \"0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754\": rpc error: code = NotFound desc = could not find container \"0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754\": container with ID starting with 0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754 not found: ID does not exist"
Sep 17 22:19:22.217507 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.217478    2567 scope.go:117] "RemoveContainer" containerID="55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0"
Sep 17 22:19:22.217707 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:19:22.217689    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0\": container with ID starting with 55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0 not found: ID does not exist" containerID="55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0"
Sep 17 22:19:22.217745 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.217713    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0"} err="failed to get container status \"55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0\": rpc error: code = NotFound desc = could not find container \"55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0\": container with ID starting with 55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0 not found: ID does not exist"
Sep 17 22:19:22.217745 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.217729    2567 scope.go:117] "RemoveContainer" containerID="0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754"
Sep 17 22:19:22.217941 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.217924    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754"} err="failed to get container status \"0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754\": rpc error: code = NotFound desc = could not find container \"0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754\": container with ID starting with 0271eaadd232b74cca4f5d5e4f240e48c2acb3e929c67186f951c3cbfad0f754 not found: ID does not exist"
Sep 17 22:19:22.217989 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.217941    2567 scope.go:117] "RemoveContainer" containerID="55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0"
Sep 17 22:19:22.218168 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.218153    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0"} err="failed to get container status \"55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0\": rpc error: code = NotFound desc = could not find container \"55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0\": container with ID starting with 55e51054a3b321d5eb8ee3df47d16baa2454acef1c98c53a1b0893e53163e4e0 not found: ID does not exist"
Sep 17 22:19:22.225276 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.225256    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"]
Sep 17 22:19:22.229174 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:22.229156    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8grm8"]
Sep 17 22:19:23.640076 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:19:23.640048    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58be1ae1-a5e1-490c-a0ae-4602a8c22092" path="/var/lib/kubelet/pods/58be1ae1-a5e1-490c-a0ae-4602a8c22092/volumes"
Sep 17 22:20:23.268883 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.268849    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"]
Sep 17 22:20:23.271255 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.269126    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58be1ae1-a5e1-490c-a0ae-4602a8c22092" containerName="main"
Sep 17 22:20:23.271255 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.269137    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="58be1ae1-a5e1-490c-a0ae-4602a8c22092" containerName="main"
Sep 17 22:20:23.271255 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.269145    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58be1ae1-a5e1-490c-a0ae-4602a8c22092" containerName="llm-d-routing-sidecar"
Sep 17 22:20:23.271255 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.269150    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="58be1ae1-a5e1-490c-a0ae-4602a8c22092" containerName="llm-d-routing-sidecar"
Sep 17 22:20:23.271255 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.269191    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="58be1ae1-a5e1-490c-a0ae-4602a8c22092" containerName="llm-d-routing-sidecar"
Sep 17 22:20:23.271255 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.269200    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="58be1ae1-a5e1-490c-a0ae-4602a8c22092" containerName="main"
Sep 17 22:20:23.272266 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.272250    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.274657 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.274631    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 22:20:23.274772 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.274675    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 22:20:23.275475 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.275455    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 22:20:23.275602 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.275583    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 22:20:23.275666 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.275624    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:20:23.275666 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.275638    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-tn5p7\""
Sep 17 22:20:23.285010 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.284982    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"]
Sep 17 22:20:23.327356 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.327326    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.327481 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.327360    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-model-cache\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.327481 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.327393    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-home\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.327481 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.327436    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-dshm\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.327481 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.327453    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbf9a25-da86-4e0a-a280-402b6911e48d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.327481 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.327467    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddbf9a25-da86-4e0a-a280-402b6911e48d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.327652 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.327503    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.327652 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.327540    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss5sr\" (UniqueName: \"kubernetes.io/projected/ddbf9a25-da86-4e0a-a280-402b6911e48d-kube-api-access-ss5sr\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.427986 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.427956    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbf9a25-da86-4e0a-a280-402b6911e48d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.427986 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.427988    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddbf9a25-da86-4e0a-a280-402b6911e48d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.428202 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.428035    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.428202 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.428080    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss5sr\" (UniqueName: \"kubernetes.io/projected/ddbf9a25-da86-4e0a-a280-402b6911e48d-kube-api-access-ss5sr\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.428202 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.428144    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.428202 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.428175    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-model-cache\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.428406 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.428223    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-home\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.428406 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.428261    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-dshm\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.428625 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.428600    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.428750 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.428684    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.428750 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.428740    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-model-cache\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.428839 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.428753    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-home\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.428839 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.428778    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddbf9a25-da86-4e0a-a280-402b6911e48d-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.430519 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.430500    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-dshm\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.430581 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.430543    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbf9a25-da86-4e0a-a280-402b6911e48d-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.436095 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.436075    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss5sr\" (UniqueName: \"kubernetes.io/projected/ddbf9a25-da86-4e0a-a280-402b6911e48d-kube-api-access-ss5sr\") pod \"tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.543052 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.542954    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"]
Sep 17 22:20:23.546842 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.546812    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"
Sep 17 22:20:23.549159 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.549139    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-bft99\""
Sep 17 22:20:23.555429 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.555408    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"]
Sep 17 22:20:23.582008 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.581980    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:23.630419 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.630388    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d202c4-d02b-42fa-96bb-5f1723ce0ee5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd\" (UID: \"a8d202c4-d02b-42fa-96bb-5f1723ce0ee5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"
Sep 17 22:20:23.630544 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.630474    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd9nl\" (UniqueName: \"kubernetes.io/projected/a8d202c4-d02b-42fa-96bb-5f1723ce0ee5-kube-api-access-nd9nl\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd\" (UID: \"a8d202c4-d02b-42fa-96bb-5f1723ce0ee5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"
Sep 17 22:20:23.731089 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.731056    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d202c4-d02b-42fa-96bb-5f1723ce0ee5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd\" (UID: \"a8d202c4-d02b-42fa-96bb-5f1723ce0ee5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"
Sep 17 22:20:23.731251 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.731114    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd9nl\" (UniqueName: \"kubernetes.io/projected/a8d202c4-d02b-42fa-96bb-5f1723ce0ee5-kube-api-access-nd9nl\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd\" (UID: \"a8d202c4-d02b-42fa-96bb-5f1723ce0ee5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"
Sep 17 22:20:23.733620 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.733596    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d202c4-d02b-42fa-96bb-5f1723ce0ee5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd\" (UID: \"a8d202c4-d02b-42fa-96bb-5f1723ce0ee5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"
Sep 17 22:20:23.738364 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.738340    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd9nl\" (UniqueName: \"kubernetes.io/projected/a8d202c4-d02b-42fa-96bb-5f1723ce0ee5-kube-api-access-nd9nl\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd\" (UID: \"a8d202c4-d02b-42fa-96bb-5f1723ce0ee5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"
Sep 17 22:20:23.850986 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.850963    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"]
Sep 17 22:20:23.853452 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:20:23.853430    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddbf9a25_da86_4e0a_a280_402b6911e48d.slice/crio-e5c628c0a9e76ec9b0f3681853affa370a6bd94bed9f0674800713eb52cea6c7 WatchSource:0}: Error finding container e5c628c0a9e76ec9b0f3681853affa370a6bd94bed9f0674800713eb52cea6c7: Status 404 returned error can't find the container with id e5c628c0a9e76ec9b0f3681853affa370a6bd94bed9f0674800713eb52cea6c7
Sep 17 22:20:23.855328 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.855312    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:20:23.857888 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:23.857868    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"
Sep 17 22:20:24.124496 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:24.124473    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"]
Sep 17 22:20:24.126834 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:20:24.126808    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d202c4_d02b_42fa_96bb_5f1723ce0ee5.slice/crio-5e9d4a0c155339cfdae0e9135c68ff598ec331a9c1024049ea02a064198e2f32 WatchSource:0}: Error finding container 5e9d4a0c155339cfdae0e9135c68ff598ec331a9c1024049ea02a064198e2f32: Status 404 returned error can't find the container with id 5e9d4a0c155339cfdae0e9135c68ff598ec331a9c1024049ea02a064198e2f32
Sep 17 22:20:24.435579 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:24.435500    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b" event={"ID":"ddbf9a25-da86-4e0a-a280-402b6911e48d","Type":"ContainerStarted","Data":"e5c628c0a9e76ec9b0f3681853affa370a6bd94bed9f0674800713eb52cea6c7"}
Sep 17 22:20:24.439579 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:24.439539    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd" event={"ID":"a8d202c4-d02b-42fa-96bb-5f1723ce0ee5","Type":"ContainerStarted","Data":"802f5805aee461f0d8fcbcf71ce4d559fece0dd1b2cc2199a4b2da8495612fb9"}
Sep 17 22:20:24.439721 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:24.439601    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd" event={"ID":"a8d202c4-d02b-42fa-96bb-5f1723ce0ee5","Type":"ContainerStarted","Data":"5e9d4a0c155339cfdae0e9135c68ff598ec331a9c1024049ea02a064198e2f32"}
Sep 17 22:20:24.439721 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:24.439646    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"
Sep 17 22:20:24.458341 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:24.458207    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd" podStartSLOduration=1.458188505 podStartE2EDuration="1.458188505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:20:24.456240858 +0000 UTC m=+1301.465170401" watchObservedRunningTime="2026-09-17 22:20:24.458188505 +0000 UTC m=+1301.467118048"
Sep 17 22:20:28.459754 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:28.459700    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b" event={"ID":"ddbf9a25-da86-4e0a-a280-402b6911e48d","Type":"ContainerStarted","Data":"d37a3fcb2651417241816b1722bc015120f5da367931db1fc835651b7a815184"}
Sep 17 22:20:30.468106 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:30.468073    2567 generic.go:356] "Generic (PLEG): container finished" podID="ddbf9a25-da86-4e0a-a280-402b6911e48d" containerID="d37a3fcb2651417241816b1722bc015120f5da367931db1fc835651b7a815184" exitCode=0
Sep 17 22:20:30.468642 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:30.468122    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b" event={"ID":"ddbf9a25-da86-4e0a-a280-402b6911e48d","Type":"ContainerDied","Data":"d37a3fcb2651417241816b1722bc015120f5da367931db1fc835651b7a815184"}
Sep 17 22:20:31.474066 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:31.474030    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b" event={"ID":"ddbf9a25-da86-4e0a-a280-402b6911e48d","Type":"ContainerStarted","Data":"376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1"}
Sep 17 22:20:31.494557 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:31.494507    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b" podStartSLOduration=2.7931149079999997 podStartE2EDuration="8.494493383s" totalImagesPullingTime="3.688441668s" totalInitContainerRuntime="2.012936807s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:20:31.493737769 +0000 UTC m=+1308.502667311" watchObservedRunningTime="2026-09-17 22:20:31.494493383 +0000 UTC m=+1308.503422924"
Sep 17 22:20:31.582798 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:31.582773    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:33.582687 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:33.582657    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:33.594805 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:33.594783    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:34.496827 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:34.496794    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:20:55.445580 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:20:55.445548    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"
Sep 17 22:21:26.360620 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:26.360549    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"]
Sep 17 22:21:28.919805 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:28.919769    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc" podUID="3ce68e03-8697-4388-8c36-f3d3f59ad634" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:21:38.919502 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:38.919461    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc" podUID="3ce68e03-8697-4388-8c36-f3d3f59ad634" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:21:41.379545 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:41.379500    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc" podUID="3ce68e03-8697-4388-8c36-f3d3f59ad634" containerName="main" containerID="cri-o://97affdd481102b7d3ce9cc5ba8173f7bed6cae85d723c61ca9bd6360a2c5f234" gracePeriod=45
Sep 17 22:21:48.920255 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:48.920214    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc" podUID="3ce68e03-8697-4388-8c36-f3d3f59ad634" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:21:48.920726 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:48.920309    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"
Sep 17 22:21:49.063325 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.063294    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"]
Sep 17 22:21:49.067360 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.067331    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"]
Sep 17 22:21:49.073032 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:21:49.072989    2567 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:21:49.073032 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:21:49.073032 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:21:49.073032 ip-10-0-142-197 kubenswrapper[2567]:  > containerID="376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1" cmd=["/bin/sleep","15"]
Sep 17 22:21:49.073299 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.073053    2567 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b" message="" err=<
Sep 17 22:21:49.073299 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:21:49.073299 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:21:49.073299 ip-10-0-142-197 kubenswrapper[2567]:  >
Sep 17 22:21:49.073299 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:21:49.073071    2567 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:21:49.073299 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:21:49.073299 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:21:49.073299 ip-10-0-142-197 kubenswrapper[2567]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b" podUID="ddbf9a25-da86-4e0a-a280-402b6911e48d" containerName="main" containerID="cri-o://376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1"
Sep 17 22:21:49.073299 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.073111    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b" podUID="ddbf9a25-da86-4e0a-a280-402b6911e48d" containerName="main" containerID="cri-o://376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1" gracePeriod=60
Sep 17 22:21:49.412758 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.412733    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:21:49.450671 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.450612    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbf9a25-da86-4e0a-a280-402b6911e48d-tls-certs\") pod \"ddbf9a25-da86-4e0a-a280-402b6911e48d\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") "
Sep 17 22:21:49.450671 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.450641    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-tmp-dir\") pod \"ddbf9a25-da86-4e0a-a280-402b6911e48d\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") "
Sep 17 22:21:49.450845 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.450674    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-dshm\") pod \"ddbf9a25-da86-4e0a-a280-402b6911e48d\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") "
Sep 17 22:21:49.450845 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.450692    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-home\") pod \"ddbf9a25-da86-4e0a-a280-402b6911e48d\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") "
Sep 17 22:21:49.450845 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.450810    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss5sr\" (UniqueName: \"kubernetes.io/projected/ddbf9a25-da86-4e0a-a280-402b6911e48d-kube-api-access-ss5sr\") pod \"ddbf9a25-da86-4e0a-a280-402b6911e48d\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") "
Sep 17 22:21:49.451017 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.450849    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddbf9a25-da86-4e0a-a280-402b6911e48d-cabundle-cert\") pod \"ddbf9a25-da86-4e0a-a280-402b6911e48d\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") "
Sep 17 22:21:49.451017 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.450883    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-kserve-provision-location\") pod \"ddbf9a25-da86-4e0a-a280-402b6911e48d\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") "
Sep 17 22:21:49.451017 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.450898    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-home" (OuterVolumeSpecName: "home") pod "ddbf9a25-da86-4e0a-a280-402b6911e48d" (UID: "ddbf9a25-da86-4e0a-a280-402b6911e48d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:49.451017 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.450918    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-model-cache\") pod \"ddbf9a25-da86-4e0a-a280-402b6911e48d\" (UID: \"ddbf9a25-da86-4e0a-a280-402b6911e48d\") "
Sep 17 22:21:49.451017 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.450924    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ddbf9a25-da86-4e0a-a280-402b6911e48d" (UID: "ddbf9a25-da86-4e0a-a280-402b6911e48d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:49.451264 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.451130    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ddbf9a25-da86-4e0a-a280-402b6911e48d" (UID: "ddbf9a25-da86-4e0a-a280-402b6911e48d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:49.451264 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.451163    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddbf9a25-da86-4e0a-a280-402b6911e48d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ddbf9a25-da86-4e0a-a280-402b6911e48d" (UID: "ddbf9a25-da86-4e0a-a280-402b6911e48d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:21:49.451264 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.451255    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.451408 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.451271    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.451408 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.451283    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.451408 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.451294    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddbf9a25-da86-4e0a-a280-402b6911e48d-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.452838 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.452803    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddbf9a25-da86-4e0a-a280-402b6911e48d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddbf9a25-da86-4e0a-a280-402b6911e48d" (UID: "ddbf9a25-da86-4e0a-a280-402b6911e48d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:21:49.453030 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.452988    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-dshm" (OuterVolumeSpecName: "dshm") pod "ddbf9a25-da86-4e0a-a280-402b6911e48d" (UID: "ddbf9a25-da86-4e0a-a280-402b6911e48d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:49.453316 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.453275    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddbf9a25-da86-4e0a-a280-402b6911e48d-kube-api-access-ss5sr" (OuterVolumeSpecName: "kube-api-access-ss5sr") pod "ddbf9a25-da86-4e0a-a280-402b6911e48d" (UID: "ddbf9a25-da86-4e0a-a280-402b6911e48d"). InnerVolumeSpecName "kube-api-access-ss5sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:21:49.470150 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.470125    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ddbf9a25-da86-4e0a-a280-402b6911e48d" (UID: "ddbf9a25-da86-4e0a-a280-402b6911e48d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:49.552103 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.552080    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddbf9a25-da86-4e0a-a280-402b6911e48d-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.552103 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.552101    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.552227 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.552110    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss5sr\" (UniqueName: \"kubernetes.io/projected/ddbf9a25-da86-4e0a-a280-402b6911e48d-kube-api-access-ss5sr\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.552227 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.552120    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddbf9a25-da86-4e0a-a280-402b6911e48d-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:21:49.767972 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.767946    2567 generic.go:356] "Generic (PLEG): container finished" podID="ddbf9a25-da86-4e0a-a280-402b6911e48d" containerID="376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1" exitCode=0
Sep 17 22:21:49.768124 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.768011    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"
Sep 17 22:21:49.768124 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.768038    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b" event={"ID":"ddbf9a25-da86-4e0a-a280-402b6911e48d","Type":"ContainerDied","Data":"376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1"}
Sep 17 22:21:49.768124 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.768094    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b" event={"ID":"ddbf9a25-da86-4e0a-a280-402b6911e48d","Type":"ContainerDied","Data":"e5c628c0a9e76ec9b0f3681853affa370a6bd94bed9f0674800713eb52cea6c7"}
Sep 17 22:21:49.768124 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.768123    2567 scope.go:117] "RemoveContainer" containerID="376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1"
Sep 17 22:21:49.778084 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.778026    2567 scope.go:117] "RemoveContainer" containerID="d37a3fcb2651417241816b1722bc015120f5da367931db1fc835651b7a815184"
Sep 17 22:21:49.783972 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.783951    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"]
Sep 17 22:21:49.786119 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.786106    2567 scope.go:117] "RemoveContainer" containerID="376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1"
Sep 17 22:21:49.786370 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:21:49.786353    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1\": container with ID starting with 376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1 not found: ID does not exist" containerID="376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1"
Sep 17 22:21:49.786432 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.786375    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1"} err="failed to get container status \"376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1\": rpc error: code = NotFound desc = could not find container \"376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1\": container with ID starting with 376ecba2428e91e997f553215b93a477cd989217e6b39935765f4e67702e7fa1 not found: ID does not exist"
Sep 17 22:21:49.786432 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.786390    2567 scope.go:117] "RemoveContainer" containerID="d37a3fcb2651417241816b1722bc015120f5da367931db1fc835651b7a815184"
Sep 17 22:21:49.786616 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:21:49.786601    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37a3fcb2651417241816b1722bc015120f5da367931db1fc835651b7a815184\": container with ID starting with d37a3fcb2651417241816b1722bc015120f5da367931db1fc835651b7a815184 not found: ID does not exist" containerID="d37a3fcb2651417241816b1722bc015120f5da367931db1fc835651b7a815184"
Sep 17 22:21:49.786653 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.786619    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37a3fcb2651417241816b1722bc015120f5da367931db1fc835651b7a815184"} err="failed to get container status \"d37a3fcb2651417241816b1722bc015120f5da367931db1fc835651b7a815184\": rpc error: code = NotFound desc = could not find container \"d37a3fcb2651417241816b1722bc015120f5da367931db1fc835651b7a815184\": container with ID starting with d37a3fcb2651417241816b1722bc015120f5da367931db1fc835651b7a815184 not found: ID does not exist"
Sep 17 22:21:49.789379 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:49.789351    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5755d67bf5-rcq4b"]
Sep 17 22:21:51.640164 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:51.640128    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddbf9a25-da86-4e0a-a280-402b6911e48d" path="/var/lib/kubelet/pods/ddbf9a25-da86-4e0a-a280-402b6911e48d/volumes"
Sep 17 22:21:55.446223 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:55.446187    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd" podUID="a8d202c4-d02b-42fa-96bb-5f1723ce0ee5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:21:58.919855 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:21:58.919822    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc" podUID="3ce68e03-8697-4388-8c36-f3d3f59ad634" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:22:04.089426 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:04.089361    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd" podUID="a8d202c4-d02b-42fa-96bb-5f1723ce0ee5" containerName="main" containerID="cri-o://802f5805aee461f0d8fcbcf71ce4d559fece0dd1b2cc2199a4b2da8495612fb9" gracePeriod=45
Sep 17 22:22:05.445427 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:05.445390    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd" podUID="a8d202c4-d02b-42fa-96bb-5f1723ce0ee5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:22:08.919596 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:08.919561    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc" podUID="3ce68e03-8697-4388-8c36-f3d3f59ad634" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:22:11.610362 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.610339    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"
Sep 17 22:22:11.704700 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.704626    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce68e03-8697-4388-8c36-f3d3f59ad634-tls-certs\") pod \"3ce68e03-8697-4388-8c36-f3d3f59ad634\" (UID: \"3ce68e03-8697-4388-8c36-f3d3f59ad634\") "
Sep 17 22:22:11.704700 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.704665    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz2k2\" (UniqueName: \"kubernetes.io/projected/3ce68e03-8697-4388-8c36-f3d3f59ad634-kube-api-access-jz2k2\") pod \"3ce68e03-8697-4388-8c36-f3d3f59ad634\" (UID: \"3ce68e03-8697-4388-8c36-f3d3f59ad634\") "
Sep 17 22:22:11.706722 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.706695    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce68e03-8697-4388-8c36-f3d3f59ad634-kube-api-access-jz2k2" (OuterVolumeSpecName: "kube-api-access-jz2k2") pod "3ce68e03-8697-4388-8c36-f3d3f59ad634" (UID: "3ce68e03-8697-4388-8c36-f3d3f59ad634"). InnerVolumeSpecName "kube-api-access-jz2k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:22:11.706819 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.706745    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce68e03-8697-4388-8c36-f3d3f59ad634-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ce68e03-8697-4388-8c36-f3d3f59ad634" (UID: "3ce68e03-8697-4388-8c36-f3d3f59ad634"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:22:11.806024 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.805986    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ce68e03-8697-4388-8c36-f3d3f59ad634-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:22:11.806024 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.806024    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz2k2\" (UniqueName: \"kubernetes.io/projected/3ce68e03-8697-4388-8c36-f3d3f59ad634-kube-api-access-jz2k2\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:22:11.854250 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.854222    2567 generic.go:356] "Generic (PLEG): container finished" podID="3ce68e03-8697-4388-8c36-f3d3f59ad634" containerID="97affdd481102b7d3ce9cc5ba8173f7bed6cae85d723c61ca9bd6360a2c5f234" exitCode=0
Sep 17 22:22:11.854361 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.854253    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc" event={"ID":"3ce68e03-8697-4388-8c36-f3d3f59ad634","Type":"ContainerDied","Data":"97affdd481102b7d3ce9cc5ba8173f7bed6cae85d723c61ca9bd6360a2c5f234"}
Sep 17 22:22:11.854361 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.854274    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"
Sep 17 22:22:11.854361 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.854307    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc" event={"ID":"3ce68e03-8697-4388-8c36-f3d3f59ad634","Type":"ContainerDied","Data":"18c8895b5236e2cd134c6055343bbda70d242c699b2410513172ef8c743781d5"}
Sep 17 22:22:11.854361 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.854333    2567 scope.go:117] "RemoveContainer" containerID="97affdd481102b7d3ce9cc5ba8173f7bed6cae85d723c61ca9bd6360a2c5f234"
Sep 17 22:22:11.862480 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.862464    2567 scope.go:117] "RemoveContainer" containerID="97affdd481102b7d3ce9cc5ba8173f7bed6cae85d723c61ca9bd6360a2c5f234"
Sep 17 22:22:11.862743 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:22:11.862726    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97affdd481102b7d3ce9cc5ba8173f7bed6cae85d723c61ca9bd6360a2c5f234\": container with ID starting with 97affdd481102b7d3ce9cc5ba8173f7bed6cae85d723c61ca9bd6360a2c5f234 not found: ID does not exist" containerID="97affdd481102b7d3ce9cc5ba8173f7bed6cae85d723c61ca9bd6360a2c5f234"
Sep 17 22:22:11.862813 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.862749    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97affdd481102b7d3ce9cc5ba8173f7bed6cae85d723c61ca9bd6360a2c5f234"} err="failed to get container status \"97affdd481102b7d3ce9cc5ba8173f7bed6cae85d723c61ca9bd6360a2c5f234\": rpc error: code = NotFound desc = could not find container \"97affdd481102b7d3ce9cc5ba8173f7bed6cae85d723c61ca9bd6360a2c5f234\": container with ID starting with 97affdd481102b7d3ce9cc5ba8173f7bed6cae85d723c61ca9bd6360a2c5f234 not found: ID does not exist"
Sep 17 22:22:11.874609 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.874588    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"]
Sep 17 22:22:11.877688 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:11.877665    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x57vc"]
Sep 17 22:22:13.639470 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:13.639439    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ce68e03-8697-4388-8c36-f3d3f59ad634" path="/var/lib/kubelet/pods/3ce68e03-8697-4388-8c36-f3d3f59ad634/volumes"
Sep 17 22:22:15.446050 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:15.446012    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd" podUID="a8d202c4-d02b-42fa-96bb-5f1723ce0ee5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:22:15.446416 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:15.446127    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"
Sep 17 22:22:25.446148 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:25.446111    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd" podUID="a8d202c4-d02b-42fa-96bb-5f1723ce0ee5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:22:26.921987 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.921951    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"]
Sep 17 22:22:26.922429 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.922347    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddbf9a25-da86-4e0a-a280-402b6911e48d" containerName="storage-initializer"
Sep 17 22:22:26.922429 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.922365    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddbf9a25-da86-4e0a-a280-402b6911e48d" containerName="storage-initializer"
Sep 17 22:22:26.922429 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.922391    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce68e03-8697-4388-8c36-f3d3f59ad634" containerName="main"
Sep 17 22:22:26.922429 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.922398    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce68e03-8697-4388-8c36-f3d3f59ad634" containerName="main"
Sep 17 22:22:26.922429 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.922412    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddbf9a25-da86-4e0a-a280-402b6911e48d" containerName="main"
Sep 17 22:22:26.922429 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.922419    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddbf9a25-da86-4e0a-a280-402b6911e48d" containerName="main"
Sep 17 22:22:26.922715 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.922481    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddbf9a25-da86-4e0a-a280-402b6911e48d" containerName="main"
Sep 17 22:22:26.922715 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.922494    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ce68e03-8697-4388-8c36-f3d3f59ad634" containerName="main"
Sep 17 22:22:26.925424 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.925402    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"
Sep 17 22:22:26.928657 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.928639    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-gn7t2\""
Sep 17 22:22:26.928761 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.928652    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-wj8t9\""
Sep 17 22:22:26.928761 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.928673    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 22:22:26.928761 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.928734    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 22:22:26.928903 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.928833    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 22:22:26.933414 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:26.933386    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"]
Sep 17 22:22:27.004952 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:27.004925    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb62d\" (UniqueName: \"kubernetes.io/projected/db4e5121-55fe-4751-aaae-14fc06b9dcbd-kube-api-access-jb62d\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98\" (UID: \"db4e5121-55fe-4751-aaae-14fc06b9dcbd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"
Sep 17 22:22:27.005109 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:27.004986    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4e5121-55fe-4751-aaae-14fc06b9dcbd-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98\" (UID: \"db4e5121-55fe-4751-aaae-14fc06b9dcbd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"
Sep 17 22:22:27.105776 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:27.105747    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb62d\" (UniqueName: \"kubernetes.io/projected/db4e5121-55fe-4751-aaae-14fc06b9dcbd-kube-api-access-jb62d\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98\" (UID: \"db4e5121-55fe-4751-aaae-14fc06b9dcbd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"
Sep 17 22:22:27.105897 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:27.105798    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4e5121-55fe-4751-aaae-14fc06b9dcbd-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98\" (UID: \"db4e5121-55fe-4751-aaae-14fc06b9dcbd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"
Sep 17 22:22:27.108276 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:27.108254    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4e5121-55fe-4751-aaae-14fc06b9dcbd-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98\" (UID: \"db4e5121-55fe-4751-aaae-14fc06b9dcbd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"
Sep 17 22:22:27.114334 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:27.114316    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb62d\" (UniqueName: \"kubernetes.io/projected/db4e5121-55fe-4751-aaae-14fc06b9dcbd-kube-api-access-jb62d\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98\" (UID: \"db4e5121-55fe-4751-aaae-14fc06b9dcbd\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"
Sep 17 22:22:27.235713 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:27.235689    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"
Sep 17 22:22:27.509692 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:27.509606    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"]
Sep 17 22:22:27.513321 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:22:27.513293    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4e5121_55fe_4751_aaae_14fc06b9dcbd.slice/crio-cad0c3657b7390063ff354d925ac9968b1207cddca750f6e37a4718e53fbc738 WatchSource:0}: Error finding container cad0c3657b7390063ff354d925ac9968b1207cddca750f6e37a4718e53fbc738: Status 404 returned error can't find the container with id cad0c3657b7390063ff354d925ac9968b1207cddca750f6e37a4718e53fbc738
Sep 17 22:22:27.915410 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:27.915327    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98" event={"ID":"db4e5121-55fe-4751-aaae-14fc06b9dcbd","Type":"ContainerStarted","Data":"9f1cc566e072bb494f5efed7d424a8eca98e8274fb63ea986512e4f2ebc33b75"}
Sep 17 22:22:27.915410 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:27.915380    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98" event={"ID":"db4e5121-55fe-4751-aaae-14fc06b9dcbd","Type":"ContainerStarted","Data":"cad0c3657b7390063ff354d925ac9968b1207cddca750f6e37a4718e53fbc738"}
Sep 17 22:22:27.915581 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:27.915448    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"
Sep 17 22:22:27.933687 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:27.933631    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98" podStartSLOduration=1.933611098 podStartE2EDuration="1.933611098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:22:27.931856045 +0000 UTC m=+1424.940785586" watchObservedRunningTime="2026-09-17 22:22:27.933611098 +0000 UTC m=+1424.942540639"
Sep 17 22:22:34.329994 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.329970    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"
Sep 17 22:22:34.458433 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.458368    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d202c4-d02b-42fa-96bb-5f1723ce0ee5-tls-certs\") pod \"a8d202c4-d02b-42fa-96bb-5f1723ce0ee5\" (UID: \"a8d202c4-d02b-42fa-96bb-5f1723ce0ee5\") "
Sep 17 22:22:34.458433 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.458414    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd9nl\" (UniqueName: \"kubernetes.io/projected/a8d202c4-d02b-42fa-96bb-5f1723ce0ee5-kube-api-access-nd9nl\") pod \"a8d202c4-d02b-42fa-96bb-5f1723ce0ee5\" (UID: \"a8d202c4-d02b-42fa-96bb-5f1723ce0ee5\") "
Sep 17 22:22:34.460437 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.460409    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d202c4-d02b-42fa-96bb-5f1723ce0ee5-kube-api-access-nd9nl" (OuterVolumeSpecName: "kube-api-access-nd9nl") pod "a8d202c4-d02b-42fa-96bb-5f1723ce0ee5" (UID: "a8d202c4-d02b-42fa-96bb-5f1723ce0ee5"). InnerVolumeSpecName "kube-api-access-nd9nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:22:34.460531 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.460476    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d202c4-d02b-42fa-96bb-5f1723ce0ee5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8d202c4-d02b-42fa-96bb-5f1723ce0ee5" (UID: "a8d202c4-d02b-42fa-96bb-5f1723ce0ee5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:22:34.558967 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.558936    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8d202c4-d02b-42fa-96bb-5f1723ce0ee5-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:22:34.558967 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.558962    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd9nl\" (UniqueName: \"kubernetes.io/projected/a8d202c4-d02b-42fa-96bb-5f1723ce0ee5-kube-api-access-nd9nl\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:22:34.941932 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.941898    2567 generic.go:356] "Generic (PLEG): container finished" podID="a8d202c4-d02b-42fa-96bb-5f1723ce0ee5" containerID="802f5805aee461f0d8fcbcf71ce4d559fece0dd1b2cc2199a4b2da8495612fb9" exitCode=0
Sep 17 22:22:34.942132 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.941946    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"
Sep 17 22:22:34.942132 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.941994    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd" event={"ID":"a8d202c4-d02b-42fa-96bb-5f1723ce0ee5","Type":"ContainerDied","Data":"802f5805aee461f0d8fcbcf71ce4d559fece0dd1b2cc2199a4b2da8495612fb9"}
Sep 17 22:22:34.942132 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.942075    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd" event={"ID":"a8d202c4-d02b-42fa-96bb-5f1723ce0ee5","Type":"ContainerDied","Data":"5e9d4a0c155339cfdae0e9135c68ff598ec331a9c1024049ea02a064198e2f32"}
Sep 17 22:22:34.942132 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.942104    2567 scope.go:117] "RemoveContainer" containerID="802f5805aee461f0d8fcbcf71ce4d559fece0dd1b2cc2199a4b2da8495612fb9"
Sep 17 22:22:34.950491 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.950472    2567 scope.go:117] "RemoveContainer" containerID="802f5805aee461f0d8fcbcf71ce4d559fece0dd1b2cc2199a4b2da8495612fb9"
Sep 17 22:22:34.950740 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:22:34.950722    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802f5805aee461f0d8fcbcf71ce4d559fece0dd1b2cc2199a4b2da8495612fb9\": container with ID starting with 802f5805aee461f0d8fcbcf71ce4d559fece0dd1b2cc2199a4b2da8495612fb9 not found: ID does not exist" containerID="802f5805aee461f0d8fcbcf71ce4d559fece0dd1b2cc2199a4b2da8495612fb9"
Sep 17 22:22:34.950782 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.950747    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802f5805aee461f0d8fcbcf71ce4d559fece0dd1b2cc2199a4b2da8495612fb9"} err="failed to get container status \"802f5805aee461f0d8fcbcf71ce4d559fece0dd1b2cc2199a4b2da8495612fb9\": rpc error: code = NotFound desc = could not find container \"802f5805aee461f0d8fcbcf71ce4d559fece0dd1b2cc2199a4b2da8495612fb9\": container with ID starting with 802f5805aee461f0d8fcbcf71ce4d559fece0dd1b2cc2199a4b2da8495612fb9 not found: ID does not exist"
Sep 17 22:22:34.962600 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.962574    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"]
Sep 17 22:22:34.966341 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:34.966320    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69qzgkd"]
Sep 17 22:22:35.639175 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:35.639144    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8d202c4-d02b-42fa-96bb-5f1723ce0ee5" path="/var/lib/kubelet/pods/a8d202c4-d02b-42fa-96bb-5f1723ce0ee5/volumes"
Sep 17 22:22:58.921163 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:22:58.921134    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"
Sep 17 22:23:01.945824 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:01.945791    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"]
Sep 17 22:23:01.946253 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:01.946076    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8d202c4-d02b-42fa-96bb-5f1723ce0ee5" containerName="main"
Sep 17 22:23:01.946253 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:01.946088    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8d202c4-d02b-42fa-96bb-5f1723ce0ee5" containerName="main"
Sep 17 22:23:01.946253 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:01.946135    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8d202c4-d02b-42fa-96bb-5f1723ce0ee5" containerName="main"
Sep 17 22:23:01.948850 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:01.948787    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:01.951146 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:01.951120    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 22:23:01.951508 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:01.951490    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 22:23:01.951624 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:01.951494    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 22:23:01.952143 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:01.952127    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:23:01.952221 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:01.952141    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 22:23:01.952221 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:01.952147    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-qg2mv\""
Sep 17 22:23:01.963152 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:01.963131    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"]
Sep 17 22:23:02.041700 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.041664    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-dshm\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.041700 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.041698    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-tls-certs\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.041907 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.041798    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-home\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.041907 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.041824    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-model-cache\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.041907 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.041862    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.041907 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.041890    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.042077 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.041908    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.042077 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.041930    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4bmc\" (UniqueName: \"kubernetes.io/projected/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-kube-api-access-w4bmc\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.142331 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.142288    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.142331 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.142329    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.142566 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.142354    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4bmc\" (UniqueName: \"kubernetes.io/projected/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-kube-api-access-w4bmc\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.142566 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.142379    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-dshm\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.142566 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.142405    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-tls-certs\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.142712 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.142572    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-home\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.142712 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.142617    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-model-cache\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.142712 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.142675    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.142865 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.142786    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.143069 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.142984    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-home\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.143069 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.143022    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.143217 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.143067    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-model-cache\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.143217 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.143173    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.144648 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.144629    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-dshm\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.144899 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.144882    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-tls-certs\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.149702 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.149678    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4bmc\" (UniqueName: \"kubernetes.io/projected/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-kube-api-access-w4bmc\") pod \"tokenizer-migration-path-test-kserve-867557464b-9hlcg\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.230099 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.230072    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"]
Sep 17 22:23:02.233966 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.233944    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"
Sep 17 22:23:02.236303 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.236281    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-mgx2b\""
Sep 17 22:23:02.241102 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.241080    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"]
Sep 17 22:23:02.259189 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.259158    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:02.344606 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.344571    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2klm\" (UniqueName: \"kubernetes.io/projected/77b3631e-8db6-47ba-ba8b-fc49dd5126f5-kube-api-access-q2klm\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts\" (UID: \"77b3631e-8db6-47ba-ba8b-fc49dd5126f5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"
Sep 17 22:23:02.344775 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.344693    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b3631e-8db6-47ba-ba8b-fc49dd5126f5-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts\" (UID: \"77b3631e-8db6-47ba-ba8b-fc49dd5126f5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"
Sep 17 22:23:02.445138 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.445104    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b3631e-8db6-47ba-ba8b-fc49dd5126f5-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts\" (UID: \"77b3631e-8db6-47ba-ba8b-fc49dd5126f5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"
Sep 17 22:23:02.445269 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.445165    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2klm\" (UniqueName: \"kubernetes.io/projected/77b3631e-8db6-47ba-ba8b-fc49dd5126f5-kube-api-access-q2klm\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts\" (UID: \"77b3631e-8db6-47ba-ba8b-fc49dd5126f5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"
Sep 17 22:23:02.447702 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.447676    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b3631e-8db6-47ba-ba8b-fc49dd5126f5-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts\" (UID: \"77b3631e-8db6-47ba-ba8b-fc49dd5126f5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"
Sep 17 22:23:02.453889 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.453866    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2klm\" (UniqueName: \"kubernetes.io/projected/77b3631e-8db6-47ba-ba8b-fc49dd5126f5-kube-api-access-q2klm\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts\" (UID: \"77b3631e-8db6-47ba-ba8b-fc49dd5126f5\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"
Sep 17 22:23:02.530272 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.530246    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"]
Sep 17 22:23:02.532431 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:23:02.532403    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25c5e57_ba5d_47a1_91c1_2a88e2f1c3de.slice/crio-ab850b2ed4cb162528687ec44a6202499cf84b2034275517c018421789c20261 WatchSource:0}: Error finding container ab850b2ed4cb162528687ec44a6202499cf84b2034275517c018421789c20261: Status 404 returned error can't find the container with id ab850b2ed4cb162528687ec44a6202499cf84b2034275517c018421789c20261
Sep 17 22:23:02.544862 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.544838    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"
Sep 17 22:23:02.816797 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:02.816767    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"]
Sep 17 22:23:02.819120 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:23:02.819095    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b3631e_8db6_47ba_ba8b_fc49dd5126f5.slice/crio-44b9e3ad6be3edde14333361c771a2de54a48a563e105ee623dd48fbd32821ca WatchSource:0}: Error finding container 44b9e3ad6be3edde14333361c771a2de54a48a563e105ee623dd48fbd32821ca: Status 404 returned error can't find the container with id 44b9e3ad6be3edde14333361c771a2de54a48a563e105ee623dd48fbd32821ca
Sep 17 22:23:03.046664 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:03.046471    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts" event={"ID":"77b3631e-8db6-47ba-ba8b-fc49dd5126f5","Type":"ContainerStarted","Data":"5ad53043c5f18d1e81be0febc0ab9fd79f990018c9a564dcd42f90310cdd859d"}
Sep 17 22:23:03.046664 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:03.046556    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts" event={"ID":"77b3631e-8db6-47ba-ba8b-fc49dd5126f5","Type":"ContainerStarted","Data":"44b9e3ad6be3edde14333361c771a2de54a48a563e105ee623dd48fbd32821ca"}
Sep 17 22:23:03.047235 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:03.046746    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"
Sep 17 22:23:03.048509 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:03.048483    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg" event={"ID":"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de","Type":"ContainerStarted","Data":"27c8dae8ef29c3509a6cf6fc73e5770d23f21386cb0099280b88c819c58efda6"}
Sep 17 22:23:03.048637 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:03.048528    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg" event={"ID":"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de","Type":"ContainerStarted","Data":"ab850b2ed4cb162528687ec44a6202499cf84b2034275517c018421789c20261"}
Sep 17 22:23:03.062862 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:03.062810    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts" podStartSLOduration=1.062796425 podStartE2EDuration="1.062796425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:23:03.061805171 +0000 UTC m=+1460.070734714" watchObservedRunningTime="2026-09-17 22:23:03.062796425 +0000 UTC m=+1460.071725958"
Sep 17 22:23:05.058828 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:05.058748    2567 generic.go:356] "Generic (PLEG): container finished" podID="b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" containerID="27c8dae8ef29c3509a6cf6fc73e5770d23f21386cb0099280b88c819c58efda6" exitCode=0
Sep 17 22:23:05.058828 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:05.058806    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg" event={"ID":"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de","Type":"ContainerDied","Data":"27c8dae8ef29c3509a6cf6fc73e5770d23f21386cb0099280b88c819c58efda6"}
Sep 17 22:23:06.065100 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:06.065065    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg" event={"ID":"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de","Type":"ContainerStarted","Data":"e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070"}
Sep 17 22:23:06.086297 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:06.086250    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg" podStartSLOduration=2.93581593 podStartE2EDuration="5.086235317s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.150419387s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:23:06.084562857 +0000 UTC m=+1463.093492414" watchObservedRunningTime="2026-09-17 22:23:06.086235317 +0000 UTC m=+1463.095164859"
Sep 17 22:23:06.259439 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:06.259408    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:12.260099 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:12.260064    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:12.272149 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:12.272127    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:13.104804 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:13.104775    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:34.055120 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:34.055091    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"
Sep 17 22:23:40.144573 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:40.144540    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"]
Sep 17 22:23:40.147922 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:40.147898    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"]
Sep 17 22:23:40.153427 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:23:40.153393    2567 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:23:40.153427 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:23:40.153427 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:23:40.153427 ip-10-0-142-197 kubenswrapper[2567]:  > containerID="e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070" cmd=["/bin/sleep","15"]
Sep 17 22:23:40.153660 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:40.153445    2567 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg" message="" err=<
Sep 17 22:23:40.153660 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:23:40.153660 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:23:40.153660 ip-10-0-142-197 kubenswrapper[2567]:  >
Sep 17 22:23:40.153660 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:23:40.153462    2567 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:23:40.153660 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:23:40.153660 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:23:40.153660 ip-10-0-142-197 kubenswrapper[2567]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg" podUID="b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" containerName="main" containerID="cri-o://e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070"
Sep 17 22:23:40.153660 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:40.153504    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg" podUID="b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" containerName="main" containerID="cri-o://e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070" gracePeriod=60
Sep 17 22:23:40.887610 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:40.887587    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:41.015819 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.015792    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-home\") pod \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") "
Sep 17 22:23:41.016019 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.015832    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-model-cache\") pod \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") "
Sep 17 22:23:41.016019 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.015863    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4bmc\" (UniqueName: \"kubernetes.io/projected/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-kube-api-access-w4bmc\") pod \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") "
Sep 17 22:23:41.016019 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.015892    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-dshm\") pod \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") "
Sep 17 22:23:41.016019 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.015914    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-cabundle-cert\") pod \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") "
Sep 17 22:23:41.016019 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.015971    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-tls-certs\") pod \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") "
Sep 17 22:23:41.016019 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.016015    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-tmp-dir\") pod \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") "
Sep 17 22:23:41.016344 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.016067    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-kserve-provision-location\") pod \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\" (UID: \"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de\") "
Sep 17 22:23:41.016344 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.016111    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-model-cache" (OuterVolumeSpecName: "model-cache") pod "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" (UID: "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:23:41.016344 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.016128    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-home" (OuterVolumeSpecName: "home") pod "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" (UID: "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:23:41.016344 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.016328    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:23:41.016488 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.016340    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" (UID: "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:23:41.016488 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.016346    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:23:41.016488 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.016385    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" (UID: "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:23:41.018072 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.018038    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" (UID: "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:23:41.018185 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.018095    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-dshm" (OuterVolumeSpecName: "dshm") pod "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" (UID: "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:23:41.018232 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.018192    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-kube-api-access-w4bmc" (OuterVolumeSpecName: "kube-api-access-w4bmc") pod "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" (UID: "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de"). InnerVolumeSpecName "kube-api-access-w4bmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:23:41.035382 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.035360    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" (UID: "b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:23:41.117055 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.117033    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:23:41.117055 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.117051    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4bmc\" (UniqueName: \"kubernetes.io/projected/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-kube-api-access-w4bmc\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:23:41.117168 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.117060    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:23:41.117168 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.117069    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:23:41.117168 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.117077    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:23:41.117168 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.117084    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:23:41.197748 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.197718    2567 generic.go:356] "Generic (PLEG): container finished" podID="b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" containerID="e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070" exitCode=0
Sep 17 22:23:41.198133 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.197789    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"
Sep 17 22:23:41.198133 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.197810    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg" event={"ID":"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de","Type":"ContainerDied","Data":"e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070"}
Sep 17 22:23:41.198133 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.197887    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg" event={"ID":"b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de","Type":"ContainerDied","Data":"ab850b2ed4cb162528687ec44a6202499cf84b2034275517c018421789c20261"}
Sep 17 22:23:41.198133 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.197916    2567 scope.go:117] "RemoveContainer" containerID="e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070"
Sep 17 22:23:41.206376 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.206354    2567 scope.go:117] "RemoveContainer" containerID="27c8dae8ef29c3509a6cf6fc73e5770d23f21386cb0099280b88c819c58efda6"
Sep 17 22:23:41.213337 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.213323    2567 scope.go:117] "RemoveContainer" containerID="e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070"
Sep 17 22:23:41.213590 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:23:41.213575    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070\": container with ID starting with e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070 not found: ID does not exist" containerID="e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070"
Sep 17 22:23:41.213626 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.213597    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070"} err="failed to get container status \"e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070\": rpc error: code = NotFound desc = could not find container \"e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070\": container with ID starting with e108d0b07799cc52bf5b1250c112f54749158f30bd04f09ead7b42ecdabf0070 not found: ID does not exist"
Sep 17 22:23:41.213626 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.213611    2567 scope.go:117] "RemoveContainer" containerID="27c8dae8ef29c3509a6cf6fc73e5770d23f21386cb0099280b88c819c58efda6"
Sep 17 22:23:41.213842 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:23:41.213819    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c8dae8ef29c3509a6cf6fc73e5770d23f21386cb0099280b88c819c58efda6\": container with ID starting with 27c8dae8ef29c3509a6cf6fc73e5770d23f21386cb0099280b88c819c58efda6 not found: ID does not exist" containerID="27c8dae8ef29c3509a6cf6fc73e5770d23f21386cb0099280b88c819c58efda6"
Sep 17 22:23:41.213921 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.213844    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c8dae8ef29c3509a6cf6fc73e5770d23f21386cb0099280b88c819c58efda6"} err="failed to get container status \"27c8dae8ef29c3509a6cf6fc73e5770d23f21386cb0099280b88c819c58efda6\": rpc error: code = NotFound desc = could not find container \"27c8dae8ef29c3509a6cf6fc73e5770d23f21386cb0099280b88c819c58efda6\": container with ID starting with 27c8dae8ef29c3509a6cf6fc73e5770d23f21386cb0099280b88c819c58efda6 not found: ID does not exist"
Sep 17 22:23:41.218070 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.218050    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"]
Sep 17 22:23:41.219923 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.219903    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-867557464b-9hlcg"]
Sep 17 22:23:41.639201 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:41.639175    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" path="/var/lib/kubelet/pods/b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de/volumes"
Sep 17 22:23:43.532799 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:43.532775    2567 scope.go:117] "RemoveContainer" containerID="b277834ba135ff1a9b1e83069d9d53c169b3391a5347cdc71df04e6ed3ccd110"
Sep 17 22:23:44.055313 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:44.055283    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts" podUID="77b3631e-8db6-47ba-ba8b-fc49dd5126f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:23:54.055627 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:54.055593    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts" podUID="77b3631e-8db6-47ba-ba8b-fc49dd5126f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:23:55.165893 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:23:55.165858    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts" podUID="77b3631e-8db6-47ba-ba8b-fc49dd5126f5" containerName="main" containerID="cri-o://5ad53043c5f18d1e81be0febc0ab9fd79f990018c9a564dcd42f90310cdd859d" gracePeriod=45
Sep 17 22:24:04.055450 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:04.055412    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts" podUID="77b3631e-8db6-47ba-ba8b-fc49dd5126f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:24:04.055803 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:04.055513    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"
Sep 17 22:24:14.055432 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:14.055351    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts" podUID="77b3631e-8db6-47ba-ba8b-fc49dd5126f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:24:24.055033 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:24.054977    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts" podUID="77b3631e-8db6-47ba-ba8b-fc49dd5126f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:24:25.353914 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:25.353885    2567 generic.go:356] "Generic (PLEG): container finished" podID="77b3631e-8db6-47ba-ba8b-fc49dd5126f5" containerID="5ad53043c5f18d1e81be0febc0ab9fd79f990018c9a564dcd42f90310cdd859d" exitCode=0
Sep 17 22:24:25.354228 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:25.353934    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts" event={"ID":"77b3631e-8db6-47ba-ba8b-fc49dd5126f5","Type":"ContainerDied","Data":"5ad53043c5f18d1e81be0febc0ab9fd79f990018c9a564dcd42f90310cdd859d"}
Sep 17 22:24:25.403500 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:25.403480    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"
Sep 17 22:24:25.504673 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:25.504646    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2klm\" (UniqueName: \"kubernetes.io/projected/77b3631e-8db6-47ba-ba8b-fc49dd5126f5-kube-api-access-q2klm\") pod \"77b3631e-8db6-47ba-ba8b-fc49dd5126f5\" (UID: \"77b3631e-8db6-47ba-ba8b-fc49dd5126f5\") "
Sep 17 22:24:25.504788 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:25.504702    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b3631e-8db6-47ba-ba8b-fc49dd5126f5-tls-certs\") pod \"77b3631e-8db6-47ba-ba8b-fc49dd5126f5\" (UID: \"77b3631e-8db6-47ba-ba8b-fc49dd5126f5\") "
Sep 17 22:24:25.506650 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:25.506627    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77b3631e-8db6-47ba-ba8b-fc49dd5126f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77b3631e-8db6-47ba-ba8b-fc49dd5126f5" (UID: "77b3631e-8db6-47ba-ba8b-fc49dd5126f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:24:25.506730 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:25.506626    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b3631e-8db6-47ba-ba8b-fc49dd5126f5-kube-api-access-q2klm" (OuterVolumeSpecName: "kube-api-access-q2klm") pod "77b3631e-8db6-47ba-ba8b-fc49dd5126f5" (UID: "77b3631e-8db6-47ba-ba8b-fc49dd5126f5"). InnerVolumeSpecName "kube-api-access-q2klm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:24:25.605075 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:25.605052    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b3631e-8db6-47ba-ba8b-fc49dd5126f5-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:24:25.605075 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:25.605073    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2klm\" (UniqueName: \"kubernetes.io/projected/77b3631e-8db6-47ba-ba8b-fc49dd5126f5-kube-api-access-q2klm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:24:26.358502 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:26.358464    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts" event={"ID":"77b3631e-8db6-47ba-ba8b-fc49dd5126f5","Type":"ContainerDied","Data":"44b9e3ad6be3edde14333361c771a2de54a48a563e105ee623dd48fbd32821ca"}
Sep 17 22:24:26.358909 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:26.358529    2567 scope.go:117] "RemoveContainer" containerID="5ad53043c5f18d1e81be0febc0ab9fd79f990018c9a564dcd42f90310cdd859d"
Sep 17 22:24:26.358909 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:26.358490    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"
Sep 17 22:24:26.374223 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:26.374203    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"]
Sep 17 22:24:26.377813 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:26.377790    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785wrwts"]
Sep 17 22:24:27.639906 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:27.639872    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77b3631e-8db6-47ba-ba8b-fc49dd5126f5" path="/var/lib/kubelet/pods/77b3631e-8db6-47ba-ba8b-fc49dd5126f5/volumes"
Sep 17 22:24:28.572082 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:28.572043    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"]
Sep 17 22:24:28.921816 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:28.921735    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98" podUID="db4e5121-55fe-4751-aaae-14fc06b9dcbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:24:33.767956 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.767926    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"]
Sep 17 22:24:33.768355 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.768242    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77b3631e-8db6-47ba-ba8b-fc49dd5126f5" containerName="main"
Sep 17 22:24:33.768355 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.768257    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="77b3631e-8db6-47ba-ba8b-fc49dd5126f5" containerName="main"
Sep 17 22:24:33.768355 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.768269    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" containerName="storage-initializer"
Sep 17 22:24:33.768355 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.768275    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" containerName="storage-initializer"
Sep 17 22:24:33.768355 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.768286    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" containerName="main"
Sep 17 22:24:33.768355 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.768294    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" containerName="main"
Sep 17 22:24:33.768355 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.768344    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="77b3631e-8db6-47ba-ba8b-fc49dd5126f5" containerName="main"
Sep 17 22:24:33.768355 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.768354    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25c5e57-ba5d-47a1-91c1-2a88e2f1c3de" containerName="main"
Sep 17 22:24:33.772667 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.772646    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.774829 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.774806    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 22:24:33.775087 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.775070    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 22:24:33.775870 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.775848    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-z26p2\""
Sep 17 22:24:33.776012 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.775851    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:24:33.776012 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.775854    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 22:24:33.776012 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.775871    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 22:24:33.785397 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.785377    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"]
Sep 17 22:24:33.862404 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.862371    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-home\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.862558 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.862422    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.862558 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.862474    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pmz6\" (UniqueName: \"kubernetes.io/projected/ce8504e8-3766-407f-815e-123064f077a4-kube-api-access-8pmz6\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.862558 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.862512    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce8504e8-3766-407f-815e-123064f077a4-tls-certs\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.862558 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.862551    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.862691 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.862575    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-model-cache\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.862691 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.862601    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-dshm\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.862691 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.862632    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8504e8-3766-407f-815e-123064f077a4-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.963565 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.963535    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8504e8-3766-407f-815e-123064f077a4-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.963565 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.963567    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-home\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.963720 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.963592    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.963720 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.963616    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmz6\" (UniqueName: \"kubernetes.io/projected/ce8504e8-3766-407f-815e-123064f077a4-kube-api-access-8pmz6\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.963720 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.963645    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce8504e8-3766-407f-815e-123064f077a4-tls-certs\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.963720 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.963679    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.963720 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.963711    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-model-cache\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.963950 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.963734    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-dshm\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.964038 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.963982    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.964096 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.964058    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-home\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.964163 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.964146    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.964230 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.964207    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-model-cache\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.964389 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.964370    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8504e8-3766-407f-815e-123064f077a4-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.965860 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.965840    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-dshm\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.966014 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.965985    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce8504e8-3766-407f-815e-123064f077a4-tls-certs\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:33.973084 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:33.973064    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmz6\" (UniqueName: \"kubernetes.io/projected/ce8504e8-3766-407f-815e-123064f077a4-kube-api-access-8pmz6\") pod \"tokenizer-template-args-test-kserve-5b44b88946-xz7q9\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:34.048028 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.047951    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"]
Sep 17 22:24:34.052108 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.052090    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"
Sep 17 22:24:34.054217 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.054195    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-bsc6w\""
Sep 17 22:24:34.061037 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.061012    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"]
Sep 17 22:24:34.083727 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.083701    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:34.165832 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.165804    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb1882d-416e-4aa2-8e76-651dbcb94e7f-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8\" (UID: \"0fb1882d-416e-4aa2-8e76-651dbcb94e7f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"
Sep 17 22:24:34.165955 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.165881    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqb92\" (UniqueName: \"kubernetes.io/projected/0fb1882d-416e-4aa2-8e76-651dbcb94e7f-kube-api-access-pqb92\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8\" (UID: \"0fb1882d-416e-4aa2-8e76-651dbcb94e7f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"
Sep 17 22:24:34.267006 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.266978    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqb92\" (UniqueName: \"kubernetes.io/projected/0fb1882d-416e-4aa2-8e76-651dbcb94e7f-kube-api-access-pqb92\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8\" (UID: \"0fb1882d-416e-4aa2-8e76-651dbcb94e7f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"
Sep 17 22:24:34.267111 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.267043    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb1882d-416e-4aa2-8e76-651dbcb94e7f-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8\" (UID: \"0fb1882d-416e-4aa2-8e76-651dbcb94e7f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"
Sep 17 22:24:34.269317 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.269297    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb1882d-416e-4aa2-8e76-651dbcb94e7f-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8\" (UID: \"0fb1882d-416e-4aa2-8e76-651dbcb94e7f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"
Sep 17 22:24:34.275286 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.275268    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqb92\" (UniqueName: \"kubernetes.io/projected/0fb1882d-416e-4aa2-8e76-651dbcb94e7f-kube-api-access-pqb92\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8\" (UID: \"0fb1882d-416e-4aa2-8e76-651dbcb94e7f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"
Sep 17 22:24:34.355011 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.354978    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"]
Sep 17 22:24:34.356915 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:24:34.356894    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8504e8_3766_407f_815e_123064f077a4.slice/crio-7a790599bc0b8c51ecb0406c1352bb1f377cb5696d36243476c814b76a2a7a41 WatchSource:0}: Error finding container 7a790599bc0b8c51ecb0406c1352bb1f377cb5696d36243476c814b76a2a7a41: Status 404 returned error can't find the container with id 7a790599bc0b8c51ecb0406c1352bb1f377cb5696d36243476c814b76a2a7a41
Sep 17 22:24:34.362547 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.362528    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"
Sep 17 22:24:34.390601 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.390579    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9" event={"ID":"ce8504e8-3766-407f-815e-123064f077a4","Type":"ContainerStarted","Data":"7a790599bc0b8c51ecb0406c1352bb1f377cb5696d36243476c814b76a2a7a41"}
Sep 17 22:24:34.631860 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:34.631729    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"]
Sep 17 22:24:34.634457 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:24:34.634432    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb1882d_416e_4aa2_8e76_651dbcb94e7f.slice/crio-6e82442ab082fc0833246410e4281b16b85e7b75d982baa67278ec4adadfd8f6 WatchSource:0}: Error finding container 6e82442ab082fc0833246410e4281b16b85e7b75d982baa67278ec4adadfd8f6: Status 404 returned error can't find the container with id 6e82442ab082fc0833246410e4281b16b85e7b75d982baa67278ec4adadfd8f6
Sep 17 22:24:35.396133 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:35.396094    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9" event={"ID":"ce8504e8-3766-407f-815e-123064f077a4","Type":"ContainerStarted","Data":"d415b65dbaa90a94330c1ea84c1fcef6be435dec8c1572514eebc0dbff8d397d"}
Sep 17 22:24:35.397795 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:35.397768    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8" event={"ID":"0fb1882d-416e-4aa2-8e76-651dbcb94e7f","Type":"ContainerStarted","Data":"f6c1861f5933459f65be245192eea60e362773ac37f783fec7f429f74abe8513"}
Sep 17 22:24:35.397926 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:35.397809    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8" event={"ID":"0fb1882d-416e-4aa2-8e76-651dbcb94e7f","Type":"ContainerStarted","Data":"6e82442ab082fc0833246410e4281b16b85e7b75d982baa67278ec4adadfd8f6"}
Sep 17 22:24:35.397926 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:35.397856    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"
Sep 17 22:24:35.430882 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:35.430835    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8" podStartSLOduration=1.4308193359999999 podStartE2EDuration="1.430819336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:24:35.430358883 +0000 UTC m=+1552.439288453" watchObservedRunningTime="2026-09-17 22:24:35.430819336 +0000 UTC m=+1552.439748881"
Sep 17 22:24:37.408556 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:37.408521    2567 generic.go:356] "Generic (PLEG): container finished" podID="ce8504e8-3766-407f-815e-123064f077a4" containerID="d415b65dbaa90a94330c1ea84c1fcef6be435dec8c1572514eebc0dbff8d397d" exitCode=0
Sep 17 22:24:37.408920 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:37.408600    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9" event={"ID":"ce8504e8-3766-407f-815e-123064f077a4","Type":"ContainerDied","Data":"d415b65dbaa90a94330c1ea84c1fcef6be435dec8c1572514eebc0dbff8d397d"}
Sep 17 22:24:38.414212 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:38.414178    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9" event={"ID":"ce8504e8-3766-407f-815e-123064f077a4","Type":"ContainerStarted","Data":"df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c"}
Sep 17 22:24:38.433367 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:38.433320    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9" podStartSLOduration=3.3417798149999998 podStartE2EDuration="5.433302487s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.091522672s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:24:38.432300175 +0000 UTC m=+1555.441229716" watchObservedRunningTime="2026-09-17 22:24:38.433302487 +0000 UTC m=+1555.442232028"
Sep 17 22:24:38.921054 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:38.921022    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98" podUID="db4e5121-55fe-4751-aaae-14fc06b9dcbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:24:39.084170 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:39.084133    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:43.590306 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:43.590271    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98" podUID="db4e5121-55fe-4751-aaae-14fc06b9dcbd" containerName="main" containerID="cri-o://9f1cc566e072bb494f5efed7d424a8eca98e8274fb63ea986512e4f2ebc33b75" gracePeriod=45
Sep 17 22:24:44.084821 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:44.084791    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:44.097231 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:44.097209    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:44.451780 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:44.451699    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:24:48.921487 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:48.921448    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98" podUID="db4e5121-55fe-4751-aaae-14fc06b9dcbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:24:48.922041 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:48.921567    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"
Sep 17 22:24:58.920900 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:24:58.920864    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98" podUID="db4e5121-55fe-4751-aaae-14fc06b9dcbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:25:06.405687 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:06.405654    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"
Sep 17 22:25:08.921295 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:08.921261    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98" podUID="db4e5121-55fe-4751-aaae-14fc06b9dcbd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:25:11.755543 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:11.755505    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"]
Sep 17 22:25:11.768028 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:11.767985    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"]
Sep 17 22:25:11.776772 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:25:11.776729    2567 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:25:11.776772 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:25:11.776772 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:25:11.776772 ip-10-0-142-197 kubenswrapper[2567]:  > containerID="df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c" cmd=["/bin/sleep","15"]
Sep 17 22:25:11.777182 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:11.776794    2567 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9" message="" err=<
Sep 17 22:25:11.777182 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:25:11.777182 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:25:11.777182 ip-10-0-142-197 kubenswrapper[2567]:  >
Sep 17 22:25:11.777182 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:25:11.776813    2567 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:25:11.777182 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:25:11.777182 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:25:11.777182 ip-10-0-142-197 kubenswrapper[2567]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9" podUID="ce8504e8-3766-407f-815e-123064f077a4" containerName="main" containerID="cri-o://df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c"
Sep 17 22:25:11.777182 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:11.776855    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9" podUID="ce8504e8-3766-407f-815e-123064f077a4" containerName="main" containerID="cri-o://df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c" gracePeriod=60
Sep 17 22:25:12.108749 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.108726    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:25:12.251760 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.251733    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8504e8-3766-407f-815e-123064f077a4-cabundle-cert\") pod \"ce8504e8-3766-407f-815e-123064f077a4\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") "
Sep 17 22:25:12.251885 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.251769    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-model-cache\") pod \"ce8504e8-3766-407f-815e-123064f077a4\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") "
Sep 17 22:25:12.251885 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.251787    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-home\") pod \"ce8504e8-3766-407f-815e-123064f077a4\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") "
Sep 17 22:25:12.251885 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.251806    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-dshm\") pod \"ce8504e8-3766-407f-815e-123064f077a4\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") "
Sep 17 22:25:12.251982 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.251933    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-kserve-provision-location\") pod \"ce8504e8-3766-407f-815e-123064f077a4\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") "
Sep 17 22:25:12.252053 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.251994    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce8504e8-3766-407f-815e-123064f077a4-tls-certs\") pod \"ce8504e8-3766-407f-815e-123064f077a4\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") "
Sep 17 22:25:12.252098 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.252074    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pmz6\" (UniqueName: \"kubernetes.io/projected/ce8504e8-3766-407f-815e-123064f077a4-kube-api-access-8pmz6\") pod \"ce8504e8-3766-407f-815e-123064f077a4\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") "
Sep 17 22:25:12.252098 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.252077    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce8504e8-3766-407f-815e-123064f077a4" (UID: "ce8504e8-3766-407f-815e-123064f077a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:12.252207 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.252084    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-home" (OuterVolumeSpecName: "home") pod "ce8504e8-3766-407f-815e-123064f077a4" (UID: "ce8504e8-3766-407f-815e-123064f077a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:12.252207 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.252108    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-tmp-dir\") pod \"ce8504e8-3766-407f-815e-123064f077a4\" (UID: \"ce8504e8-3766-407f-815e-123064f077a4\") "
Sep 17 22:25:12.252207 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.252199    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8504e8-3766-407f-815e-123064f077a4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce8504e8-3766-407f-815e-123064f077a4" (UID: "ce8504e8-3766-407f-815e-123064f077a4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:25:12.252438 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.252405    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce8504e8-3766-407f-815e-123064f077a4" (UID: "ce8504e8-3766-407f-815e-123064f077a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:12.252438 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.252429    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8504e8-3766-407f-815e-123064f077a4-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.252554 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.252447    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.252554 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.252456    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.254114 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.254086    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-dshm" (OuterVolumeSpecName: "dshm") pod "ce8504e8-3766-407f-815e-123064f077a4" (UID: "ce8504e8-3766-407f-815e-123064f077a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:12.254207 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.254147    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8504e8-3766-407f-815e-123064f077a4-kube-api-access-8pmz6" (OuterVolumeSpecName: "kube-api-access-8pmz6") pod "ce8504e8-3766-407f-815e-123064f077a4" (UID: "ce8504e8-3766-407f-815e-123064f077a4"). InnerVolumeSpecName "kube-api-access-8pmz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:25:12.254207 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.254168    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce8504e8-3766-407f-815e-123064f077a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce8504e8-3766-407f-815e-123064f077a4" (UID: "ce8504e8-3766-407f-815e-123064f077a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:25:12.277277 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.277252    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce8504e8-3766-407f-815e-123064f077a4" (UID: "ce8504e8-3766-407f-815e-123064f077a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:25:12.353167 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.353114    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce8504e8-3766-407f-815e-123064f077a4-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.353167 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.353136    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pmz6\" (UniqueName: \"kubernetes.io/projected/ce8504e8-3766-407f-815e-123064f077a4-kube-api-access-8pmz6\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.353167 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.353147    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.353167 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.353157    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.353167 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.353164    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8504e8-3766-407f-815e-123064f077a4-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:25:12.546499 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.546453    2567 generic.go:356] "Generic (PLEG): container finished" podID="ce8504e8-3766-407f-815e-123064f077a4" containerID="df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c" exitCode=0
Sep 17 22:25:12.546663 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.546522    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"
Sep 17 22:25:12.546663 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.546536    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9" event={"ID":"ce8504e8-3766-407f-815e-123064f077a4","Type":"ContainerDied","Data":"df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c"}
Sep 17 22:25:12.546663 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.546600    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9" event={"ID":"ce8504e8-3766-407f-815e-123064f077a4","Type":"ContainerDied","Data":"7a790599bc0b8c51ecb0406c1352bb1f377cb5696d36243476c814b76a2a7a41"}
Sep 17 22:25:12.546663 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.546631    2567 scope.go:117] "RemoveContainer" containerID="df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c"
Sep 17 22:25:12.556908 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.556893    2567 scope.go:117] "RemoveContainer" containerID="d415b65dbaa90a94330c1ea84c1fcef6be435dec8c1572514eebc0dbff8d397d"
Sep 17 22:25:12.563838 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.563824    2567 scope.go:117] "RemoveContainer" containerID="df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c"
Sep 17 22:25:12.564111 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:25:12.564094    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c\": container with ID starting with df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c not found: ID does not exist" containerID="df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c"
Sep 17 22:25:12.564190 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.564120    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c"} err="failed to get container status \"df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c\": rpc error: code = NotFound desc = could not find container \"df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c\": container with ID starting with df5cb84395b795193ab47a4d86db230befb0ff4d30b7c451c5b641c2c7471a5c not found: ID does not exist"
Sep 17 22:25:12.564190 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.564135    2567 scope.go:117] "RemoveContainer" containerID="d415b65dbaa90a94330c1ea84c1fcef6be435dec8c1572514eebc0dbff8d397d"
Sep 17 22:25:12.564386 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:25:12.564368    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d415b65dbaa90a94330c1ea84c1fcef6be435dec8c1572514eebc0dbff8d397d\": container with ID starting with d415b65dbaa90a94330c1ea84c1fcef6be435dec8c1572514eebc0dbff8d397d not found: ID does not exist" containerID="d415b65dbaa90a94330c1ea84c1fcef6be435dec8c1572514eebc0dbff8d397d"
Sep 17 22:25:12.564428 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.564391    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d415b65dbaa90a94330c1ea84c1fcef6be435dec8c1572514eebc0dbff8d397d"} err="failed to get container status \"d415b65dbaa90a94330c1ea84c1fcef6be435dec8c1572514eebc0dbff8d397d\": rpc error: code = NotFound desc = could not find container \"d415b65dbaa90a94330c1ea84c1fcef6be435dec8c1572514eebc0dbff8d397d\": container with ID starting with d415b65dbaa90a94330c1ea84c1fcef6be435dec8c1572514eebc0dbff8d397d not found: ID does not exist"
Sep 17 22:25:12.570230 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.570199    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"]
Sep 17 22:25:12.575599 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:12.575581    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b44b88946-xz7q9"]
Sep 17 22:25:13.639765 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:13.639729    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce8504e8-3766-407f-815e-123064f077a4" path="/var/lib/kubelet/pods/ce8504e8-3766-407f-815e-123064f077a4/volumes"
Sep 17 22:25:13.830285 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:13.830262    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"
Sep 17 22:25:13.965354 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:13.965286    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4e5121-55fe-4751-aaae-14fc06b9dcbd-tls-certs\") pod \"db4e5121-55fe-4751-aaae-14fc06b9dcbd\" (UID: \"db4e5121-55fe-4751-aaae-14fc06b9dcbd\") "
Sep 17 22:25:13.965354 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:13.965345    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb62d\" (UniqueName: \"kubernetes.io/projected/db4e5121-55fe-4751-aaae-14fc06b9dcbd-kube-api-access-jb62d\") pod \"db4e5121-55fe-4751-aaae-14fc06b9dcbd\" (UID: \"db4e5121-55fe-4751-aaae-14fc06b9dcbd\") "
Sep 17 22:25:13.967281 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:13.967256    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4e5121-55fe-4751-aaae-14fc06b9dcbd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db4e5121-55fe-4751-aaae-14fc06b9dcbd" (UID: "db4e5121-55fe-4751-aaae-14fc06b9dcbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:25:13.967360 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:13.967327    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4e5121-55fe-4751-aaae-14fc06b9dcbd-kube-api-access-jb62d" (OuterVolumeSpecName: "kube-api-access-jb62d") pod "db4e5121-55fe-4751-aaae-14fc06b9dcbd" (UID: "db4e5121-55fe-4751-aaae-14fc06b9dcbd"). InnerVolumeSpecName "kube-api-access-jb62d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:25:14.066433 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:14.066406    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb62d\" (UniqueName: \"kubernetes.io/projected/db4e5121-55fe-4751-aaae-14fc06b9dcbd-kube-api-access-jb62d\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:25:14.066433 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:14.066431    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db4e5121-55fe-4751-aaae-14fc06b9dcbd-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:25:14.555933 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:14.555902    2567 generic.go:356] "Generic (PLEG): container finished" podID="db4e5121-55fe-4751-aaae-14fc06b9dcbd" containerID="9f1cc566e072bb494f5efed7d424a8eca98e8274fb63ea986512e4f2ebc33b75" exitCode=0
Sep 17 22:25:14.556169 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:14.555931    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98" event={"ID":"db4e5121-55fe-4751-aaae-14fc06b9dcbd","Type":"ContainerDied","Data":"9f1cc566e072bb494f5efed7d424a8eca98e8274fb63ea986512e4f2ebc33b75"}
Sep 17 22:25:14.556169 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:14.555952    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"
Sep 17 22:25:14.556169 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:14.555985    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98" event={"ID":"db4e5121-55fe-4751-aaae-14fc06b9dcbd","Type":"ContainerDied","Data":"cad0c3657b7390063ff354d925ac9968b1207cddca750f6e37a4718e53fbc738"}
Sep 17 22:25:14.556169 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:14.556024    2567 scope.go:117] "RemoveContainer" containerID="9f1cc566e072bb494f5efed7d424a8eca98e8274fb63ea986512e4f2ebc33b75"
Sep 17 22:25:14.569484 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:14.569462    2567 scope.go:117] "RemoveContainer" containerID="9f1cc566e072bb494f5efed7d424a8eca98e8274fb63ea986512e4f2ebc33b75"
Sep 17 22:25:14.569841 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:25:14.569819    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1cc566e072bb494f5efed7d424a8eca98e8274fb63ea986512e4f2ebc33b75\": container with ID starting with 9f1cc566e072bb494f5efed7d424a8eca98e8274fb63ea986512e4f2ebc33b75 not found: ID does not exist" containerID="9f1cc566e072bb494f5efed7d424a8eca98e8274fb63ea986512e4f2ebc33b75"
Sep 17 22:25:14.569913 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:14.569850    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1cc566e072bb494f5efed7d424a8eca98e8274fb63ea986512e4f2ebc33b75"} err="failed to get container status \"9f1cc566e072bb494f5efed7d424a8eca98e8274fb63ea986512e4f2ebc33b75\": rpc error: code = NotFound desc = could not find container \"9f1cc566e072bb494f5efed7d424a8eca98e8274fb63ea986512e4f2ebc33b75\": container with ID starting with 9f1cc566e072bb494f5efed7d424a8eca98e8274fb63ea986512e4f2ebc33b75 not found: ID does not exist"
Sep 17 22:25:14.582056 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:14.582035    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"]
Sep 17 22:25:14.588234 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:14.588215    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cg2s98"]
Sep 17 22:25:15.639402 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:15.639367    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db4e5121-55fe-4751-aaae-14fc06b9dcbd" path="/var/lib/kubelet/pods/db4e5121-55fe-4751-aaae-14fc06b9dcbd/volumes"
Sep 17 22:25:16.405506 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:16.405464    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8" podUID="0fb1882d-416e-4aa2-8e76-651dbcb94e7f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:25:24.080407 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.080377    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"]
Sep 17 22:25:24.082897 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.080631    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db4e5121-55fe-4751-aaae-14fc06b9dcbd" containerName="main"
Sep 17 22:25:24.082897 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.080642    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="db4e5121-55fe-4751-aaae-14fc06b9dcbd" containerName="main"
Sep 17 22:25:24.082897 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.080653    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce8504e8-3766-407f-815e-123064f077a4" containerName="storage-initializer"
Sep 17 22:25:24.082897 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.080659    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce8504e8-3766-407f-815e-123064f077a4" containerName="storage-initializer"
Sep 17 22:25:24.082897 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.080675    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce8504e8-3766-407f-815e-123064f077a4" containerName="main"
Sep 17 22:25:24.082897 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.080681    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce8504e8-3766-407f-815e-123064f077a4" containerName="main"
Sep 17 22:25:24.082897 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.080729    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce8504e8-3766-407f-815e-123064f077a4" containerName="main"
Sep 17 22:25:24.082897 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.080737    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="db4e5121-55fe-4751-aaae-14fc06b9dcbd" containerName="main"
Sep 17 22:25:24.083831 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.083814    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"
Sep 17 22:25:24.086415 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.086386    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 22:25:24.086589 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.086571    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 22:25:24.086660 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.086614    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-6lqz5\""
Sep 17 22:25:24.086724 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.086704    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 22:25:24.086935 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.086921    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-69g5t\""
Sep 17 22:25:24.094417 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.094398    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"]
Sep 17 22:25:24.241165 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.241128    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf29133-d18c-41c1-a67c-e55077f53787-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w\" (UID: \"dcf29133-d18c-41c1-a67c-e55077f53787\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"
Sep 17 22:25:24.241350 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.241238    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97n5z\" (UniqueName: \"kubernetes.io/projected/dcf29133-d18c-41c1-a67c-e55077f53787-kube-api-access-97n5z\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w\" (UID: \"dcf29133-d18c-41c1-a67c-e55077f53787\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"
Sep 17 22:25:24.342234 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.342134    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97n5z\" (UniqueName: \"kubernetes.io/projected/dcf29133-d18c-41c1-a67c-e55077f53787-kube-api-access-97n5z\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w\" (UID: \"dcf29133-d18c-41c1-a67c-e55077f53787\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"
Sep 17 22:25:24.342234 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.342201    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf29133-d18c-41c1-a67c-e55077f53787-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w\" (UID: \"dcf29133-d18c-41c1-a67c-e55077f53787\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"
Sep 17 22:25:24.344646 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.344619    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf29133-d18c-41c1-a67c-e55077f53787-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w\" (UID: \"dcf29133-d18c-41c1-a67c-e55077f53787\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"
Sep 17 22:25:24.351841 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.351817    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97n5z\" (UniqueName: \"kubernetes.io/projected/dcf29133-d18c-41c1-a67c-e55077f53787-kube-api-access-97n5z\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w\" (UID: \"dcf29133-d18c-41c1-a67c-e55077f53787\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"
Sep 17 22:25:24.394172 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.394149    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"
Sep 17 22:25:24.665235 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:24.665193    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"]
Sep 17 22:25:24.667774 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:25:24.667739    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf29133_d18c_41c1_a67c_e55077f53787.slice/crio-fc789906f3e187b1e8e85e55977479078131bab22aa1b3e8ba37e167d53e1835 WatchSource:0}: Error finding container fc789906f3e187b1e8e85e55977479078131bab22aa1b3e8ba37e167d53e1835: Status 404 returned error can't find the container with id fc789906f3e187b1e8e85e55977479078131bab22aa1b3e8ba37e167d53e1835
Sep 17 22:25:25.597619 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:25.597580    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w" event={"ID":"dcf29133-d18c-41c1-a67c-e55077f53787","Type":"ContainerStarted","Data":"d5c189130024c5e8abd4950fd649f65104c15a6bdd97f526c43a5728cfc7496b"}
Sep 17 22:25:25.598025 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:25.597640    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w" event={"ID":"dcf29133-d18c-41c1-a67c-e55077f53787","Type":"ContainerStarted","Data":"fc789906f3e187b1e8e85e55977479078131bab22aa1b3e8ba37e167d53e1835"}
Sep 17 22:25:25.598025 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:25.597711    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"
Sep 17 22:25:25.612377 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:25.612326    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w" podStartSLOduration=1.612312615 podStartE2EDuration="1.612312615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:25:25.612065083 +0000 UTC m=+1602.620994626" watchObservedRunningTime="2026-09-17 22:25:25.612312615 +0000 UTC m=+1602.621242156"
Sep 17 22:25:26.404978 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:26.404939    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8" podUID="0fb1882d-416e-4aa2-8e76-651dbcb94e7f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:25:26.775044 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:26.774987    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8" podUID="0fb1882d-416e-4aa2-8e76-651dbcb94e7f" containerName="main" containerID="cri-o://f6c1861f5933459f65be245192eea60e362773ac37f783fec7f429f74abe8513" gracePeriod=45
Sep 17 22:25:36.405424 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:36.405341    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8" podUID="0fb1882d-416e-4aa2-8e76-651dbcb94e7f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:25:36.405780 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:36.405437    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"
Sep 17 22:25:46.405147 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:46.405111    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8" podUID="0fb1882d-416e-4aa2-8e76-651dbcb94e7f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:25:56.405513 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:56.405477    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8" podUID="0fb1882d-416e-4aa2-8e76-651dbcb94e7f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:25:56.602847 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:56.602819    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"
Sep 17 22:25:57.017700 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.017677    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"
Sep 17 22:25:57.046093 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.046026    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb1882d-416e-4aa2-8e76-651dbcb94e7f-tls-certs\") pod \"0fb1882d-416e-4aa2-8e76-651dbcb94e7f\" (UID: \"0fb1882d-416e-4aa2-8e76-651dbcb94e7f\") "
Sep 17 22:25:57.046223 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.046108    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqb92\" (UniqueName: \"kubernetes.io/projected/0fb1882d-416e-4aa2-8e76-651dbcb94e7f-kube-api-access-pqb92\") pod \"0fb1882d-416e-4aa2-8e76-651dbcb94e7f\" (UID: \"0fb1882d-416e-4aa2-8e76-651dbcb94e7f\") "
Sep 17 22:25:57.048051 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.048018    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb1882d-416e-4aa2-8e76-651dbcb94e7f-kube-api-access-pqb92" (OuterVolumeSpecName: "kube-api-access-pqb92") pod "0fb1882d-416e-4aa2-8e76-651dbcb94e7f" (UID: "0fb1882d-416e-4aa2-8e76-651dbcb94e7f"). InnerVolumeSpecName "kube-api-access-pqb92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:25:57.048051 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.048046    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb1882d-416e-4aa2-8e76-651dbcb94e7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fb1882d-416e-4aa2-8e76-651dbcb94e7f" (UID: "0fb1882d-416e-4aa2-8e76-651dbcb94e7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:25:57.146685 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.146657    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqb92\" (UniqueName: \"kubernetes.io/projected/0fb1882d-416e-4aa2-8e76-651dbcb94e7f-kube-api-access-pqb92\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:25:57.146685 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.146680    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fb1882d-416e-4aa2-8e76-651dbcb94e7f-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:25:57.712687 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.712651    2567 generic.go:356] "Generic (PLEG): container finished" podID="0fb1882d-416e-4aa2-8e76-651dbcb94e7f" containerID="f6c1861f5933459f65be245192eea60e362773ac37f783fec7f429f74abe8513" exitCode=0
Sep 17 22:25:57.713108 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.712706    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8" event={"ID":"0fb1882d-416e-4aa2-8e76-651dbcb94e7f","Type":"ContainerDied","Data":"f6c1861f5933459f65be245192eea60e362773ac37f783fec7f429f74abe8513"}
Sep 17 22:25:57.713108 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.712720    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"
Sep 17 22:25:57.713108 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.712741    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8" event={"ID":"0fb1882d-416e-4aa2-8e76-651dbcb94e7f","Type":"ContainerDied","Data":"6e82442ab082fc0833246410e4281b16b85e7b75d982baa67278ec4adadfd8f6"}
Sep 17 22:25:57.713108 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.712761    2567 scope.go:117] "RemoveContainer" containerID="f6c1861f5933459f65be245192eea60e362773ac37f783fec7f429f74abe8513"
Sep 17 22:25:57.720494 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.720477    2567 scope.go:117] "RemoveContainer" containerID="f6c1861f5933459f65be245192eea60e362773ac37f783fec7f429f74abe8513"
Sep 17 22:25:57.720732 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:25:57.720716    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c1861f5933459f65be245192eea60e362773ac37f783fec7f429f74abe8513\": container with ID starting with f6c1861f5933459f65be245192eea60e362773ac37f783fec7f429f74abe8513 not found: ID does not exist" containerID="f6c1861f5933459f65be245192eea60e362773ac37f783fec7f429f74abe8513"
Sep 17 22:25:57.720773 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.720739    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c1861f5933459f65be245192eea60e362773ac37f783fec7f429f74abe8513"} err="failed to get container status \"f6c1861f5933459f65be245192eea60e362773ac37f783fec7f429f74abe8513\": rpc error: code = NotFound desc = could not find container \"f6c1861f5933459f65be245192eea60e362773ac37f783fec7f429f74abe8513\": container with ID starting with f6c1861f5933459f65be245192eea60e362773ac37f783fec7f429f74abe8513 not found: ID does not exist"
Sep 17 22:25:57.727601 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.727580    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"]
Sep 17 22:25:57.735771 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:57.735752    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dx5gg8"]
Sep 17 22:25:59.639932 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:25:59.639899    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fb1882d-416e-4aa2-8e76-651dbcb94e7f" path="/var/lib/kubelet/pods/0fb1882d-416e-4aa2-8e76-651dbcb94e7f/volumes"
Sep 17 22:27:48.934932 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:27:48.934898    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"]
Sep 17 22:27:56.603310 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:27:56.603274    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w" podUID="dcf29133-d18c-41c1-a67c-e55077f53787" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:28:03.954576 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:03.954513    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w" podUID="dcf29133-d18c-41c1-a67c-e55077f53787" containerName="main" containerID="cri-o://d5c189130024c5e8abd4950fd649f65104c15a6bdd97f526c43a5728cfc7496b" gracePeriod=45
Sep 17 22:28:06.603302 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:06.603260    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w" podUID="dcf29133-d18c-41c1-a67c-e55077f53787" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:28:16.603555 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:16.603512    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w" podUID="dcf29133-d18c-41c1-a67c-e55077f53787" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:28:16.604121 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:16.603632    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"
Sep 17 22:28:26.603316 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:26.603280    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w" podUID="dcf29133-d18c-41c1-a67c-e55077f53787" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:28:34.192280 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.192253    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"
Sep 17 22:28:34.206083 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.206060    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97n5z\" (UniqueName: \"kubernetes.io/projected/dcf29133-d18c-41c1-a67c-e55077f53787-kube-api-access-97n5z\") pod \"dcf29133-d18c-41c1-a67c-e55077f53787\" (UID: \"dcf29133-d18c-41c1-a67c-e55077f53787\") "
Sep 17 22:28:34.206207 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.206116    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf29133-d18c-41c1-a67c-e55077f53787-tls-certs\") pod \"dcf29133-d18c-41c1-a67c-e55077f53787\" (UID: \"dcf29133-d18c-41c1-a67c-e55077f53787\") "
Sep 17 22:28:34.208696 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.208618    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf29133-d18c-41c1-a67c-e55077f53787-kube-api-access-97n5z" (OuterVolumeSpecName: "kube-api-access-97n5z") pod "dcf29133-d18c-41c1-a67c-e55077f53787" (UID: "dcf29133-d18c-41c1-a67c-e55077f53787"). InnerVolumeSpecName "kube-api-access-97n5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:28:34.208975 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.208942    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf29133-d18c-41c1-a67c-e55077f53787-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dcf29133-d18c-41c1-a67c-e55077f53787" (UID: "dcf29133-d18c-41c1-a67c-e55077f53787"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:28:34.255993 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.255962    2567 generic.go:356] "Generic (PLEG): container finished" podID="dcf29133-d18c-41c1-a67c-e55077f53787" containerID="d5c189130024c5e8abd4950fd649f65104c15a6bdd97f526c43a5728cfc7496b" exitCode=0
Sep 17 22:28:34.256112 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.256028    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"
Sep 17 22:28:34.256112 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.256047    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w" event={"ID":"dcf29133-d18c-41c1-a67c-e55077f53787","Type":"ContainerDied","Data":"d5c189130024c5e8abd4950fd649f65104c15a6bdd97f526c43a5728cfc7496b"}
Sep 17 22:28:34.256112 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.256097    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w" event={"ID":"dcf29133-d18c-41c1-a67c-e55077f53787","Type":"ContainerDied","Data":"fc789906f3e187b1e8e85e55977479078131bab22aa1b3e8ba37e167d53e1835"}
Sep 17 22:28:34.256226 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.256129    2567 scope.go:117] "RemoveContainer" containerID="d5c189130024c5e8abd4950fd649f65104c15a6bdd97f526c43a5728cfc7496b"
Sep 17 22:28:34.264088 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.264068    2567 scope.go:117] "RemoveContainer" containerID="d5c189130024c5e8abd4950fd649f65104c15a6bdd97f526c43a5728cfc7496b"
Sep 17 22:28:34.264364 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:28:34.264343    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c189130024c5e8abd4950fd649f65104c15a6bdd97f526c43a5728cfc7496b\": container with ID starting with d5c189130024c5e8abd4950fd649f65104c15a6bdd97f526c43a5728cfc7496b not found: ID does not exist" containerID="d5c189130024c5e8abd4950fd649f65104c15a6bdd97f526c43a5728cfc7496b"
Sep 17 22:28:34.264420 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.264375    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c189130024c5e8abd4950fd649f65104c15a6bdd97f526c43a5728cfc7496b"} err="failed to get container status \"d5c189130024c5e8abd4950fd649f65104c15a6bdd97f526c43a5728cfc7496b\": rpc error: code = NotFound desc = could not find container \"d5c189130024c5e8abd4950fd649f65104c15a6bdd97f526c43a5728cfc7496b\": container with ID starting with d5c189130024c5e8abd4950fd649f65104c15a6bdd97f526c43a5728cfc7496b not found: ID does not exist"
Sep 17 22:28:34.277254 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.277235    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"]
Sep 17 22:28:34.281125 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.281104    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-79x5w"]
Sep 17 22:28:34.307239 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.307216    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf29133-d18c-41c1-a67c-e55077f53787-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:28:34.307239 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.307237    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97n5z\" (UniqueName: \"kubernetes.io/projected/dcf29133-d18c-41c1-a67c-e55077f53787-kube-api-access-97n5z\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:28:34.796346 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.796312    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"]
Sep 17 22:28:34.796569 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.796557    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fb1882d-416e-4aa2-8e76-651dbcb94e7f" containerName="main"
Sep 17 22:28:34.796613 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.796570    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fb1882d-416e-4aa2-8e76-651dbcb94e7f" containerName="main"
Sep 17 22:28:34.796613 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.796577    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf29133-d18c-41c1-a67c-e55077f53787" containerName="main"
Sep 17 22:28:34.796613 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.796582    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf29133-d18c-41c1-a67c-e55077f53787" containerName="main"
Sep 17 22:28:34.796704 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.796630    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf29133-d18c-41c1-a67c-e55077f53787" containerName="main"
Sep 17 22:28:34.796704 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.796638    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fb1882d-416e-4aa2-8e76-651dbcb94e7f" containerName="main"
Sep 17 22:28:34.800767 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.800742    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.804178 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.804156    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-vmxx6\""
Sep 17 22:28:34.804296 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.804206    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 22:28:34.804843 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.804822    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 17 22:28:34.807291 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.807275    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 22:28:34.810266 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.810244    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-istio-data\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.810370 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.810278    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-workload-certs\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.810370 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.810299    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441b28b8-8966-4f47-a27d-190852d571d8-istio-token\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.810370 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.810324    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqrl6\" (UniqueName: \"kubernetes.io/projected/441b28b8-8966-4f47-a27d-190852d571d8-kube-api-access-lqrl6\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.810370 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.810361    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441b28b8-8966-4f47-a27d-190852d571d8-istio-podinfo\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.810537 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.810388    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-workload-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.810537 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.810425    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441b28b8-8966-4f47-a27d-190852d571d8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.810537 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.810447    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-istio-envoy\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.810537 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.810466    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-credential-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.821563 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.821543    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"]
Sep 17 22:28:34.911130 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.911108    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441b28b8-8966-4f47-a27d-190852d571d8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.911266 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.911139    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-istio-envoy\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.911266 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.911158    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-credential-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.911266 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.911209    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-istio-data\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.911266 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.911228    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-workload-certs\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.911266 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.911241    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441b28b8-8966-4f47-a27d-190852d571d8-istio-token\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.911266 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.911256    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqrl6\" (UniqueName: \"kubernetes.io/projected/441b28b8-8966-4f47-a27d-190852d571d8-kube-api-access-lqrl6\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.911609 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.911287    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441b28b8-8966-4f47-a27d-190852d571d8-istio-podinfo\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.911609 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.911304    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-workload-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.911609 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.911601    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-credential-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.911755 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.911726    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-workload-certs\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.911755 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.911748    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-workload-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.911860 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.911822    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441b28b8-8966-4f47-a27d-190852d571d8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.911942 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.911923    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-istio-data\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.913461 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.913440    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-istio-envoy\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.913794 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.913776    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441b28b8-8966-4f47-a27d-190852d571d8-istio-podinfo\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.922813 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.922787    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441b28b8-8966-4f47-a27d-190852d571d8-istio-token\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:34.923690 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:34.923673    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqrl6\" (UniqueName: \"kubernetes.io/projected/441b28b8-8966-4f47-a27d-190852d571d8-kube-api-access-lqrl6\") pod \"router-gateway-1-openshift-default-8555f5d76c-cmktw\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:35.110894 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:35.110844    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:35.380595 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:35.380520    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"]
Sep 17 22:28:35.384051 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:28:35.384023    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441b28b8_8966_4f47_a27d_190852d571d8.slice/crio-0236784c10b9539273f357104b4a018d360542ec16b2bab07bdf00b637e9a9bb WatchSource:0}: Error finding container 0236784c10b9539273f357104b4a018d360542ec16b2bab07bdf00b637e9a9bb: Status 404 returned error can't find the container with id 0236784c10b9539273f357104b4a018d360542ec16b2bab07bdf00b637e9a9bb
Sep 17 22:28:35.386026 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:35.385984    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:28:35.639388 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:35.639311    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcf29133-d18c-41c1-a67c-e55077f53787" path="/var/lib/kubelet/pods/dcf29133-d18c-41c1-a67c-e55077f53787/volumes"
Sep 17 22:28:36.265067 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:36.265029    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw" event={"ID":"441b28b8-8966-4f47-a27d-190852d571d8","Type":"ContainerStarted","Data":"0236784c10b9539273f357104b4a018d360542ec16b2bab07bdf00b637e9a9bb"}
Sep 17 22:28:38.548253 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:38.548209    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:28:38.548524 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:38.548305    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:28:38.548524 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:38.548338    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:28:39.279161 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:39.279123    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw" event={"ID":"441b28b8-8966-4f47-a27d-190852d571d8","Type":"ContainerStarted","Data":"0967b1c5e2bbfafb9b8db1d8ea08699fe9b59ed933d91230cc24736ee37b7095"}
Sep 17 22:28:39.298926 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:39.298881    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw" podStartSLOduration=2.137067213 podStartE2EDuration="5.298868415s" totalImagesPullingTime="3.161801202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:28:39.298347942 +0000 UTC m=+1796.307277498" watchObservedRunningTime="2026-09-17 22:28:39.298868415 +0000 UTC m=+1796.307797955"
Sep 17 22:28:40.111802 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:40.111771    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:40.116261 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:40.116240    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:40.283296 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:40.283268    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:40.284106 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:40.284089    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:45.121595 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.121564    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"]
Sep 17 22:28:45.126043 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.126018    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.130016 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.129980    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 17 22:28:45.130143 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.130026    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-5ptsq\""
Sep 17 22:28:45.134925 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.134898    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"]
Sep 17 22:28:45.188038 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.187994    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.188140 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.188056    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.188140 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.188078    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57f2340-0974-4698-bccb-d492aedd6286-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.188140 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.188098    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.188140 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.188130    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.188278 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.188151    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swh52\" (UniqueName: \"kubernetes.io/projected/c57f2340-0974-4698-bccb-d492aedd6286-kube-api-access-swh52\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.288870 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.288847    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.288995 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.288878    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.288995 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.288896    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57f2340-0974-4698-bccb-d492aedd6286-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.288995 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.288914    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.288995 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.288975    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.289235 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.289031    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swh52\" (UniqueName: \"kubernetes.io/projected/c57f2340-0974-4698-bccb-d492aedd6286-kube-api-access-swh52\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.289370 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.289334    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.289471 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.289396    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.289471 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.289421    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.291164 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.291147    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.291399 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.291383    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57f2340-0974-4698-bccb-d492aedd6286-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.296148 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.296127    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swh52\" (UniqueName: \"kubernetes.io/projected/c57f2340-0974-4698-bccb-d492aedd6286-kube-api-access-swh52\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.436395 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.436328    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:45.704080 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:45.704010    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"]
Sep 17 22:28:45.707231 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:28:45.707205    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57f2340_0974_4698_bccb_d492aedd6286.slice/crio-31c073c6920d44fa237ce21b077dde831d56edee126a2d99228e46d21b3a02b1 WatchSource:0}: Error finding container 31c073c6920d44fa237ce21b077dde831d56edee126a2d99228e46d21b3a02b1: Status 404 returned error can't find the container with id 31c073c6920d44fa237ce21b077dde831d56edee126a2d99228e46d21b3a02b1
Sep 17 22:28:46.310479 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:46.310430    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv" event={"ID":"c57f2340-0974-4698-bccb-d492aedd6286","Type":"ContainerStarted","Data":"54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a"}
Sep 17 22:28:46.310956 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:46.310518    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv" event={"ID":"c57f2340-0974-4698-bccb-d492aedd6286","Type":"ContainerStarted","Data":"31c073c6920d44fa237ce21b077dde831d56edee126a2d99228e46d21b3a02b1"}
Sep 17 22:28:46.347317 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:46.347276    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv" podStartSLOduration=1.347264057 podStartE2EDuration="1.347264057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:28:46.345012057 +0000 UTC m=+1803.353941588" watchObservedRunningTime="2026-09-17 22:28:46.347264057 +0000 UTC m=+1803.356193597"
Sep 17 22:28:46.436654 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:46.436618    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:55.285232 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:55.285195    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw" podUID="441b28b8-8966-4f47-a27d-190852d571d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 17 22:28:55.348651 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:55.348621    2567 generic.go:356] "Generic (PLEG): container finished" podID="441b28b8-8966-4f47-a27d-190852d571d8" containerID="0967b1c5e2bbfafb9b8db1d8ea08699fe9b59ed933d91230cc24736ee37b7095" exitCode=0
Sep 17 22:28:55.348786 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:55.348658    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw" event={"ID":"441b28b8-8966-4f47-a27d-190852d571d8","Type":"ContainerDied","Data":"0967b1c5e2bbfafb9b8db1d8ea08699fe9b59ed933d91230cc24736ee37b7095"}
Sep 17 22:28:55.348987 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:55.348973    2567 scope.go:117] "RemoveContainer" containerID="0967b1c5e2bbfafb9b8db1d8ea08699fe9b59ed933d91230cc24736ee37b7095"
Sep 17 22:28:55.350126 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:55.350094    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:28:55.350214 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:55.350167    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:28:55.350214 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:55.350202    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:28:55.436718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:55.436690    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:55.454507 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:55.454484    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:56.357837 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:56.357790    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw" event={"ID":"441b28b8-8966-4f47-a27d-190852d571d8","Type":"ContainerStarted","Data":"53ddf05c302fa1f48e649e7f832b2730d055a028183810e8532a3d2e1453a281"}
Sep 17 22:28:56.378684 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:56.378645    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:28:57.111900 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:57.111866    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:28:57.112125 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:57.112102    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw" podUID="441b28b8-8966-4f47-a27d-190852d571d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 17 22:28:58.111410 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:58.111364    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw" podUID="441b28b8-8966-4f47-a27d-190852d571d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 17 22:28:59.111247 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:28:59.111205    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw" podUID="441b28b8-8966-4f47-a27d-190852d571d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 17 22:29:00.111303 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:29:00.111266    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw" podUID="441b28b8-8966-4f47-a27d-190852d571d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 17 22:29:01.111895 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:29:01.111857    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw" podUID="441b28b8-8966-4f47-a27d-190852d571d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 17 22:29:02.116199 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:29:02.116166    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:29:02.116581 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:29:02.116531    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:29:02.117220 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:29:02.117203    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:30:57.217656 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.217618    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"]
Sep 17 22:30:57.220873 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.220856    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"
Sep 17 22:30:57.224200 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.224175    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 22:30:57.224320 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.224175    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 22:30:57.224320 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.224255    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-b7d4m\""
Sep 17 22:30:57.224499 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.224481    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 22:30:57.224561 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.224486    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-86gnp\""
Sep 17 22:30:57.229261 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.229233    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"]
Sep 17 22:30:57.331845 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.331817    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2kt9\" (UniqueName: \"kubernetes.io/projected/3584402f-2e79-4d0c-a469-b6d7ff0f927f-kube-api-access-c2kt9\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj\" (UID: \"3584402f-2e79-4d0c-a469-b6d7ff0f927f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"
Sep 17 22:30:57.331976 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.331854    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3584402f-2e79-4d0c-a469-b6d7ff0f927f-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj\" (UID: \"3584402f-2e79-4d0c-a469-b6d7ff0f927f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"
Sep 17 22:30:57.432500 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.432471    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3584402f-2e79-4d0c-a469-b6d7ff0f927f-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj\" (UID: \"3584402f-2e79-4d0c-a469-b6d7ff0f927f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"
Sep 17 22:30:57.432641 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.432546    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2kt9\" (UniqueName: \"kubernetes.io/projected/3584402f-2e79-4d0c-a469-b6d7ff0f927f-kube-api-access-c2kt9\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj\" (UID: \"3584402f-2e79-4d0c-a469-b6d7ff0f927f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"
Sep 17 22:30:57.434826 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.434805    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3584402f-2e79-4d0c-a469-b6d7ff0f927f-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj\" (UID: \"3584402f-2e79-4d0c-a469-b6d7ff0f927f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"
Sep 17 22:30:57.440300 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.440275    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2kt9\" (UniqueName: \"kubernetes.io/projected/3584402f-2e79-4d0c-a469-b6d7ff0f927f-kube-api-access-c2kt9\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj\" (UID: \"3584402f-2e79-4d0c-a469-b6d7ff0f927f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"
Sep 17 22:30:57.530717 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.530688    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"
Sep 17 22:30:57.797803 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:57.797725    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"]
Sep 17 22:30:57.800724 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:30:57.800700    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3584402f_2e79_4d0c_a469_b6d7ff0f927f.slice/crio-4e5675b8ee4138487d57450704ed22f1eea43b5c753c8ef6c6c10662e96d308a WatchSource:0}: Error finding container 4e5675b8ee4138487d57450704ed22f1eea43b5c753c8ef6c6c10662e96d308a: Status 404 returned error can't find the container with id 4e5675b8ee4138487d57450704ed22f1eea43b5c753c8ef6c6c10662e96d308a
Sep 17 22:30:58.800098 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:58.800042    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj" event={"ID":"3584402f-2e79-4d0c-a469-b6d7ff0f927f","Type":"ContainerStarted","Data":"82e1b82eef2947ac64fd463e2358d30be7ac83f20dfeb50d22e1814e6970f6fe"}
Sep 17 22:30:58.800498 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:58.800237    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"
Sep 17 22:30:58.800498 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:58.800270    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj" event={"ID":"3584402f-2e79-4d0c-a469-b6d7ff0f927f","Type":"ContainerStarted","Data":"4e5675b8ee4138487d57450704ed22f1eea43b5c753c8ef6c6c10662e96d308a"}
Sep 17 22:30:58.817196 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:30:58.817151    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj" podStartSLOduration=1.81713364 podStartE2EDuration="1.81713364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:30:58.815852165 +0000 UTC m=+1935.824781711" watchObservedRunningTime="2026-09-17 22:30:58.81713364 +0000 UTC m=+1935.826063183"
Sep 17 22:31:27.244311 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.244273    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"]
Sep 17 22:31:27.252760 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:31:27.252722    2567 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 22:31:27.252760 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:31:27.252760 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:31:27.252760 ip-10-0-142-197 kubenswrapper[2567]:  > containerID="54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a" cmd=["/bin/sleep","15"]
Sep 17 22:31:27.252977 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.252778    2567 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv" message="" err=<
Sep 17 22:31:27.252977 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:31:27.252977 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:31:27.252977 ip-10-0-142-197 kubenswrapper[2567]:  >
Sep 17 22:31:27.252977 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:31:27.252790    2567 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 22:31:27.252977 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 22:31:27.252977 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 22:31:27.252977 ip-10-0-142-197 kubenswrapper[2567]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv" podUID="c57f2340-0974-4698-bccb-d492aedd6286" containerName="main" containerID="cri-o://54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a"
Sep 17 22:31:27.252977 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.252827    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv" podUID="c57f2340-0974-4698-bccb-d492aedd6286" containerName="main" containerID="cri-o://54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a" gracePeriod=60
Sep 17 22:31:27.358217 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.358186    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv" podUID="c57f2340-0974-4698-bccb-d492aedd6286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 17 22:31:27.595034 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.594986    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:31:27.759768 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.759739    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57f2340-0974-4698-bccb-d492aedd6286-tls-certs\") pod \"c57f2340-0974-4698-bccb-d492aedd6286\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") "
Sep 17 22:31:27.759933 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.759774    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-dshm\") pod \"c57f2340-0974-4698-bccb-d492aedd6286\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") "
Sep 17 22:31:27.759933 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.759796    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-model-cache\") pod \"c57f2340-0974-4698-bccb-d492aedd6286\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") "
Sep 17 22:31:27.759933 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.759836    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-home\") pod \"c57f2340-0974-4698-bccb-d492aedd6286\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") "
Sep 17 22:31:27.759933 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.759866    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-tmp-dir\") pod \"c57f2340-0974-4698-bccb-d492aedd6286\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") "
Sep 17 22:31:27.759933 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.759894    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swh52\" (UniqueName: \"kubernetes.io/projected/c57f2340-0974-4698-bccb-d492aedd6286-kube-api-access-swh52\") pod \"c57f2340-0974-4698-bccb-d492aedd6286\" (UID: \"c57f2340-0974-4698-bccb-d492aedd6286\") "
Sep 17 22:31:27.760244 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.760173    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-model-cache" (OuterVolumeSpecName: "model-cache") pod "c57f2340-0974-4698-bccb-d492aedd6286" (UID: "c57f2340-0974-4698-bccb-d492aedd6286"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:31:27.760244 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.760184    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-home" (OuterVolumeSpecName: "home") pod "c57f2340-0974-4698-bccb-d492aedd6286" (UID: "c57f2340-0974-4698-bccb-d492aedd6286"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:31:27.760345 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.760245    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:27.760345 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.760263    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:27.760676 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.760650    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c57f2340-0974-4698-bccb-d492aedd6286" (UID: "c57f2340-0974-4698-bccb-d492aedd6286"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:31:27.762066 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.762046    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-dshm" (OuterVolumeSpecName: "dshm") pod "c57f2340-0974-4698-bccb-d492aedd6286" (UID: "c57f2340-0974-4698-bccb-d492aedd6286"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:31:27.762156 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.762142    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57f2340-0974-4698-bccb-d492aedd6286-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c57f2340-0974-4698-bccb-d492aedd6286" (UID: "c57f2340-0974-4698-bccb-d492aedd6286"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:31:27.762199 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.762139    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57f2340-0974-4698-bccb-d492aedd6286-kube-api-access-swh52" (OuterVolumeSpecName: "kube-api-access-swh52") pod "c57f2340-0974-4698-bccb-d492aedd6286" (UID: "c57f2340-0974-4698-bccb-d492aedd6286"). InnerVolumeSpecName "kube-api-access-swh52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:31:27.860872 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.860840    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c57f2340-0974-4698-bccb-d492aedd6286-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:27.860872 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.860866    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:27.861062 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.860880    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c57f2340-0974-4698-bccb-d492aedd6286-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:27.861062 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.860892    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swh52\" (UniqueName: \"kubernetes.io/projected/c57f2340-0974-4698-bccb-d492aedd6286-kube-api-access-swh52\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:27.911659 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.911626    2567 generic.go:356] "Generic (PLEG): container finished" podID="c57f2340-0974-4698-bccb-d492aedd6286" containerID="54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a" exitCode=0
Sep 17 22:31:27.911787 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.911671    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"
Sep 17 22:31:27.911787 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.911694    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv" event={"ID":"c57f2340-0974-4698-bccb-d492aedd6286","Type":"ContainerDied","Data":"54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a"}
Sep 17 22:31:27.911787 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.911736    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv" event={"ID":"c57f2340-0974-4698-bccb-d492aedd6286","Type":"ContainerDied","Data":"31c073c6920d44fa237ce21b077dde831d56edee126a2d99228e46d21b3a02b1"}
Sep 17 22:31:27.911787 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.911757    2567 scope.go:117] "RemoveContainer" containerID="54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a"
Sep 17 22:31:27.921881 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.921813    2567 scope.go:117] "RemoveContainer" containerID="54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a"
Sep 17 22:31:27.922126 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:31:27.922101    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a\": container with ID starting with 54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a not found: ID does not exist" containerID="54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a"
Sep 17 22:31:27.922201 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.922137    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a"} err="failed to get container status \"54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a\": rpc error: code = NotFound desc = could not find container \"54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a\": container with ID starting with 54781128ba58abf0de4ab8579535521c7a2c72eba57fe96a76d93b6100504c7a not found: ID does not exist"
Sep 17 22:31:27.932907 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.932881    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"]
Sep 17 22:31:27.938595 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:27.938575    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff589972wlbv"]
Sep 17 22:31:29.639545 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:29.639510    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c57f2340-0974-4698-bccb-d492aedd6286" path="/var/lib/kubelet/pods/c57f2340-0974-4698-bccb-d492aedd6286/volumes"
Sep 17 22:31:29.805682 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:29.805661    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"
Sep 17 22:31:35.409129 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:35.409093    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"]
Sep 17 22:31:35.409524 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:35.409314    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw" podUID="441b28b8-8966-4f47-a27d-190852d571d8" containerName="istio-proxy" containerID="cri-o://53ddf05c302fa1f48e649e7f832b2730d055a028183810e8532a3d2e1453a281" gracePeriod=30
Sep 17 22:31:40.641490 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.641465    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:31:40.755703 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.755675    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441b28b8-8966-4f47-a27d-190852d571d8-istiod-ca-cert\") pod \"441b28b8-8966-4f47-a27d-190852d571d8\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") "
Sep 17 22:31:40.755857 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.755710    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-istio-data\") pod \"441b28b8-8966-4f47-a27d-190852d571d8\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") "
Sep 17 22:31:40.755857 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.755726    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441b28b8-8966-4f47-a27d-190852d571d8-istio-podinfo\") pod \"441b28b8-8966-4f47-a27d-190852d571d8\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") "
Sep 17 22:31:40.755857 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.755743    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-workload-socket\") pod \"441b28b8-8966-4f47-a27d-190852d571d8\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") "
Sep 17 22:31:40.755857 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.755764    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-credential-socket\") pod \"441b28b8-8966-4f47-a27d-190852d571d8\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") "
Sep 17 22:31:40.755857 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.755790    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-istio-envoy\") pod \"441b28b8-8966-4f47-a27d-190852d571d8\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") "
Sep 17 22:31:40.755857 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.755843    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441b28b8-8966-4f47-a27d-190852d571d8-istio-token\") pod \"441b28b8-8966-4f47-a27d-190852d571d8\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") "
Sep 17 22:31:40.756190 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.755874    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqrl6\" (UniqueName: \"kubernetes.io/projected/441b28b8-8966-4f47-a27d-190852d571d8-kube-api-access-lqrl6\") pod \"441b28b8-8966-4f47-a27d-190852d571d8\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") "
Sep 17 22:31:40.756190 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.755926    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-workload-certs\") pod \"441b28b8-8966-4f47-a27d-190852d571d8\" (UID: \"441b28b8-8966-4f47-a27d-190852d571d8\") "
Sep 17 22:31:40.756190 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.756020    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-istio-data" (OuterVolumeSpecName: "istio-data") pod "441b28b8-8966-4f47-a27d-190852d571d8" (UID: "441b28b8-8966-4f47-a27d-190852d571d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:31:40.756190 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.756070    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "441b28b8-8966-4f47-a27d-190852d571d8" (UID: "441b28b8-8966-4f47-a27d-190852d571d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:31:40.756190 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.756070    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441b28b8-8966-4f47-a27d-190852d571d8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "441b28b8-8966-4f47-a27d-190852d571d8" (UID: "441b28b8-8966-4f47-a27d-190852d571d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:31:40.756452 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.756207    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-istio-data\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:40.756452 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.756226    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-workload-socket\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:40.756452 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.756240    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/441b28b8-8966-4f47-a27d-190852d571d8-istiod-ca-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:40.756452 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.756223    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "441b28b8-8966-4f47-a27d-190852d571d8" (UID: "441b28b8-8966-4f47-a27d-190852d571d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:31:40.756452 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.756383    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "441b28b8-8966-4f47-a27d-190852d571d8" (UID: "441b28b8-8966-4f47-a27d-190852d571d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:31:40.758222 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.758196    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441b28b8-8966-4f47-a27d-190852d571d8-kube-api-access-lqrl6" (OuterVolumeSpecName: "kube-api-access-lqrl6") pod "441b28b8-8966-4f47-a27d-190852d571d8" (UID: "441b28b8-8966-4f47-a27d-190852d571d8"). InnerVolumeSpecName "kube-api-access-lqrl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:31:40.758462 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.758442    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441b28b8-8966-4f47-a27d-190852d571d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "441b28b8-8966-4f47-a27d-190852d571d8" (UID: "441b28b8-8966-4f47-a27d-190852d571d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:31:40.758573 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.758552    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/441b28b8-8966-4f47-a27d-190852d571d8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "441b28b8-8966-4f47-a27d-190852d571d8" (UID: "441b28b8-8966-4f47-a27d-190852d571d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 22:31:40.758573 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.758553    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "441b28b8-8966-4f47-a27d-190852d571d8" (UID: "441b28b8-8966-4f47-a27d-190852d571d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:31:40.856787 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.856757    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/441b28b8-8966-4f47-a27d-190852d571d8-istio-podinfo\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:40.856787 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.856781    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-credential-socket\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:40.856787 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.856790    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-istio-envoy\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:40.856990 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.856799    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/441b28b8-8966-4f47-a27d-190852d571d8-istio-token\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:40.856990 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.856807    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqrl6\" (UniqueName: \"kubernetes.io/projected/441b28b8-8966-4f47-a27d-190852d571d8-kube-api-access-lqrl6\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:40.856990 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.856816    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/441b28b8-8966-4f47-a27d-190852d571d8-workload-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:31:40.957870 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.957792    2567 generic.go:356] "Generic (PLEG): container finished" podID="441b28b8-8966-4f47-a27d-190852d571d8" containerID="53ddf05c302fa1f48e649e7f832b2730d055a028183810e8532a3d2e1453a281" exitCode=0
Sep 17 22:31:40.957870 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.957840    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"
Sep 17 22:31:40.958059 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.957835    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw" event={"ID":"441b28b8-8966-4f47-a27d-190852d571d8","Type":"ContainerDied","Data":"53ddf05c302fa1f48e649e7f832b2730d055a028183810e8532a3d2e1453a281"}
Sep 17 22:31:40.958059 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.957950    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw" event={"ID":"441b28b8-8966-4f47-a27d-190852d571d8","Type":"ContainerDied","Data":"0236784c10b9539273f357104b4a018d360542ec16b2bab07bdf00b637e9a9bb"}
Sep 17 22:31:40.958059 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.957972    2567 scope.go:117] "RemoveContainer" containerID="53ddf05c302fa1f48e649e7f832b2730d055a028183810e8532a3d2e1453a281"
Sep 17 22:31:40.965444 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.965428    2567 scope.go:117] "RemoveContainer" containerID="0967b1c5e2bbfafb9b8db1d8ea08699fe9b59ed933d91230cc24736ee37b7095"
Sep 17 22:31:40.972363 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.972347    2567 scope.go:117] "RemoveContainer" containerID="53ddf05c302fa1f48e649e7f832b2730d055a028183810e8532a3d2e1453a281"
Sep 17 22:31:40.972621 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:31:40.972603    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ddf05c302fa1f48e649e7f832b2730d055a028183810e8532a3d2e1453a281\": container with ID starting with 53ddf05c302fa1f48e649e7f832b2730d055a028183810e8532a3d2e1453a281 not found: ID does not exist" containerID="53ddf05c302fa1f48e649e7f832b2730d055a028183810e8532a3d2e1453a281"
Sep 17 22:31:40.972685 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.972630    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ddf05c302fa1f48e649e7f832b2730d055a028183810e8532a3d2e1453a281"} err="failed to get container status \"53ddf05c302fa1f48e649e7f832b2730d055a028183810e8532a3d2e1453a281\": rpc error: code = NotFound desc = could not find container \"53ddf05c302fa1f48e649e7f832b2730d055a028183810e8532a3d2e1453a281\": container with ID starting with 53ddf05c302fa1f48e649e7f832b2730d055a028183810e8532a3d2e1453a281 not found: ID does not exist"
Sep 17 22:31:40.972685 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.972647    2567 scope.go:117] "RemoveContainer" containerID="0967b1c5e2bbfafb9b8db1d8ea08699fe9b59ed933d91230cc24736ee37b7095"
Sep 17 22:31:40.972879 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:31:40.972865    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0967b1c5e2bbfafb9b8db1d8ea08699fe9b59ed933d91230cc24736ee37b7095\": container with ID starting with 0967b1c5e2bbfafb9b8db1d8ea08699fe9b59ed933d91230cc24736ee37b7095 not found: ID does not exist" containerID="0967b1c5e2bbfafb9b8db1d8ea08699fe9b59ed933d91230cc24736ee37b7095"
Sep 17 22:31:40.972917 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.972881    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0967b1c5e2bbfafb9b8db1d8ea08699fe9b59ed933d91230cc24736ee37b7095"} err="failed to get container status \"0967b1c5e2bbfafb9b8db1d8ea08699fe9b59ed933d91230cc24736ee37b7095\": rpc error: code = NotFound desc = could not find container \"0967b1c5e2bbfafb9b8db1d8ea08699fe9b59ed933d91230cc24736ee37b7095\": container with ID starting with 0967b1c5e2bbfafb9b8db1d8ea08699fe9b59ed933d91230cc24736ee37b7095 not found: ID does not exist"
Sep 17 22:31:40.978887 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.978867    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"]
Sep 17 22:31:40.985235 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:40.985214    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-cmktw"]
Sep 17 22:31:41.063950 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.063926    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"]
Sep 17 22:31:41.064206 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.064194    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57f2340-0974-4698-bccb-d492aedd6286" containerName="main"
Sep 17 22:31:41.064250 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.064208    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57f2340-0974-4698-bccb-d492aedd6286" containerName="main"
Sep 17 22:31:41.064250 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.064224    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="441b28b8-8966-4f47-a27d-190852d571d8" containerName="istio-proxy"
Sep 17 22:31:41.064250 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.064230    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="441b28b8-8966-4f47-a27d-190852d571d8" containerName="istio-proxy"
Sep 17 22:31:41.064250 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.064241    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="441b28b8-8966-4f47-a27d-190852d571d8" containerName="istio-proxy"
Sep 17 22:31:41.064250 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.064246    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="441b28b8-8966-4f47-a27d-190852d571d8" containerName="istio-proxy"
Sep 17 22:31:41.064397 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.064300    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57f2340-0974-4698-bccb-d492aedd6286" containerName="main"
Sep 17 22:31:41.064397 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.064307    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="441b28b8-8966-4f47-a27d-190852d571d8" containerName="istio-proxy"
Sep 17 22:31:41.068310 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.068295    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"
Sep 17 22:31:41.070757 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.070732    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-hnphm\""
Sep 17 22:31:41.071056 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.071040    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 22:31:41.077984 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.077962    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"]
Sep 17 22:31:41.159047 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.159017    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42bfce2-d63b-4da3-afe0-afccb25c48cf-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5\" (UID: \"c42bfce2-d63b-4da3-afe0-afccb25c48cf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"
Sep 17 22:31:41.159173 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.159072    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwjhg\" (UniqueName: \"kubernetes.io/projected/c42bfce2-d63b-4da3-afe0-afccb25c48cf-kube-api-access-zwjhg\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5\" (UID: \"c42bfce2-d63b-4da3-afe0-afccb25c48cf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"
Sep 17 22:31:41.260203 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.260167    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwjhg\" (UniqueName: \"kubernetes.io/projected/c42bfce2-d63b-4da3-afe0-afccb25c48cf-kube-api-access-zwjhg\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5\" (UID: \"c42bfce2-d63b-4da3-afe0-afccb25c48cf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"
Sep 17 22:31:41.260355 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.260238    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42bfce2-d63b-4da3-afe0-afccb25c48cf-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5\" (UID: \"c42bfce2-d63b-4da3-afe0-afccb25c48cf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"
Sep 17 22:31:41.262561 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.262541    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42bfce2-d63b-4da3-afe0-afccb25c48cf-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5\" (UID: \"c42bfce2-d63b-4da3-afe0-afccb25c48cf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"
Sep 17 22:31:41.268394 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.268376    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwjhg\" (UniqueName: \"kubernetes.io/projected/c42bfce2-d63b-4da3-afe0-afccb25c48cf-kube-api-access-zwjhg\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5\" (UID: \"c42bfce2-d63b-4da3-afe0-afccb25c48cf\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"
Sep 17 22:31:41.378285 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.378259    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"
Sep 17 22:31:41.639398 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.639300    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="441b28b8-8966-4f47-a27d-190852d571d8" path="/var/lib/kubelet/pods/441b28b8-8966-4f47-a27d-190852d571d8/volumes"
Sep 17 22:31:41.644397 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.644183    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"]
Sep 17 22:31:41.646677 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:31:41.646653    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42bfce2_d63b_4da3_afe0_afccb25c48cf.slice/crio-14c06770e2869d950eb4d6b80bcab9fba719e00b4e1d558ebbe969651d3a4f7d WatchSource:0}: Error finding container 14c06770e2869d950eb4d6b80bcab9fba719e00b4e1d558ebbe969651d3a4f7d: Status 404 returned error can't find the container with id 14c06770e2869d950eb4d6b80bcab9fba719e00b4e1d558ebbe969651d3a4f7d
Sep 17 22:31:41.963842 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.963760    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5" event={"ID":"c42bfce2-d63b-4da3-afe0-afccb25c48cf","Type":"ContainerStarted","Data":"5614224eaf38d2cbb19fcf99f0d316f4b94dee7498d70a2c5560c6895e9bf4a4"}
Sep 17 22:31:41.963842 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.963811    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5" event={"ID":"c42bfce2-d63b-4da3-afe0-afccb25c48cf","Type":"ContainerStarted","Data":"14c06770e2869d950eb4d6b80bcab9fba719e00b4e1d558ebbe969651d3a4f7d"}
Sep 17 22:31:41.964023 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.963868    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"
Sep 17 22:31:41.980358 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:31:41.980321    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5" podStartSLOduration=0.980308046 podStartE2EDuration="980.308046ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:31:41.979334508 +0000 UTC m=+1978.988264048" watchObservedRunningTime="2026-09-17 22:31:41.980308046 +0000 UTC m=+1978.989237586"
Sep 17 22:32:10.464182 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.464148    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"]
Sep 17 22:32:10.464700 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.464610    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="441b28b8-8966-4f47-a27d-190852d571d8" containerName="istio-proxy"
Sep 17 22:32:10.467971 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.467948    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.470924 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.470898    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-b2h2t\""
Sep 17 22:32:10.471045 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.470971    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 22:32:10.471115 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.471087    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 17 22:32:10.471115 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.471096    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:32:10.471224 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.471101    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 22:32:10.472072 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.472058    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 22:32:10.481883 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.481861    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"]
Sep 17 22:32:10.571616 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.571587    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38f5997f-c74a-4fef-b3d5-4a6677576f1c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.571740 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.571630    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh48f\" (UniqueName: \"kubernetes.io/projected/38f5997f-c74a-4fef-b3d5-4a6677576f1c-kube-api-access-rh48f\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.571740 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.571672    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.571836 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.571744    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.571836 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.571765    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.571836 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.571789    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.571836 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.571807    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.571836 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.571825    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f5997f-c74a-4fef-b3d5-4a6677576f1c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.672540 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.672517    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.672693 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.672545    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.672693 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.672561    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.672693 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.672578    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.672693 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.672598    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f5997f-c74a-4fef-b3d5-4a6677576f1c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.672693 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.672669    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38f5997f-c74a-4fef-b3d5-4a6677576f1c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.672950 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.672720    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh48f\" (UniqueName: \"kubernetes.io/projected/38f5997f-c74a-4fef-b3d5-4a6677576f1c-kube-api-access-rh48f\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.672950 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.672747    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.672950 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.672927    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.673159 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.673139    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.673302 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.673195    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.673302 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.673227    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.673662 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.673646    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38f5997f-c74a-4fef-b3d5-4a6677576f1c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.675252 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.675233    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f5997f-c74a-4fef-b3d5-4a6677576f1c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.675339 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.675289    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.683466 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.683442    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh48f\" (UniqueName: \"kubernetes.io/projected/38f5997f-c74a-4fef-b3d5-4a6677576f1c-kube-api-access-rh48f\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:10.778370 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:10.778351    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:11.053501 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:11.053427    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"]
Sep 17 22:32:11.055503 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:32:11.055482    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f5997f_c74a_4fef_b3d5_4a6677576f1c.slice/crio-e89f7d3acca9ddb5cc52531f3e6a815132ba5497e0a41ddcf95010adf5484e7a WatchSource:0}: Error finding container e89f7d3acca9ddb5cc52531f3e6a815132ba5497e0a41ddcf95010adf5484e7a: Status 404 returned error can't find the container with id e89f7d3acca9ddb5cc52531f3e6a815132ba5497e0a41ddcf95010adf5484e7a
Sep 17 22:32:11.066518 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:11.066495    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" event={"ID":"38f5997f-c74a-4fef-b3d5-4a6677576f1c","Type":"ContainerStarted","Data":"e89f7d3acca9ddb5cc52531f3e6a815132ba5497e0a41ddcf95010adf5484e7a"}
Sep 17 22:32:12.072971 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:12.072935    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" event={"ID":"38f5997f-c74a-4fef-b3d5-4a6677576f1c","Type":"ContainerStarted","Data":"b18b769991ff33d55f34c6af39cbf68717b0ea673baff7c5bd08b29d81754046"}
Sep 17 22:32:12.969376 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:12.969344    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"
Sep 17 22:32:14.081056 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:14.081025    2567 generic.go:356] "Generic (PLEG): container finished" podID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerID="b18b769991ff33d55f34c6af39cbf68717b0ea673baff7c5bd08b29d81754046" exitCode=0
Sep 17 22:32:14.081056 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:14.081060    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" event={"ID":"38f5997f-c74a-4fef-b3d5-4a6677576f1c","Type":"ContainerDied","Data":"b18b769991ff33d55f34c6af39cbf68717b0ea673baff7c5bd08b29d81754046"}
Sep 17 22:32:37.543642 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:37.543608    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"]
Sep 17 22:32:42.211812 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:42.211777    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" event={"ID":"38f5997f-c74a-4fef-b3d5-4a6677576f1c","Type":"ContainerStarted","Data":"6226ec5798561285a217577be43765c9265626e4830e3b8ec62237b2920a6d7a"}
Sep 17 22:32:42.239768 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:42.239721    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" podStartSLOduration=3.05580554 podStartE2EDuration="32.239707161s" totalImagesPullingTime="27.07300305s" totalInitContainerRuntime="2.110898571s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:32:42.239315689 +0000 UTC m=+2039.248245233" watchObservedRunningTime="2026-09-17 22:32:42.239707161 +0000 UTC m=+2039.248636702"
Sep 17 22:32:42.778994 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:42.778955    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:42.969135 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:42.969096    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5" podUID="c42bfce2-d63b-4da3-afe0-afccb25c48cf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:32:50.778565 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:50.778529    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:32:50.779849 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:50.779822    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 22:32:52.568020 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:52.567959    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5" podUID="c42bfce2-d63b-4da3-afe0-afccb25c48cf" containerName="main" containerID="cri-o://5614224eaf38d2cbb19fcf99f0d316f4b94dee7498d70a2c5560c6895e9bf4a4" gracePeriod=45
Sep 17 22:32:52.969701 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:32:52.969614    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5" podUID="c42bfce2-d63b-4da3-afe0-afccb25c48cf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:33:00.779788 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:00.779741    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 22:33:02.969111 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:02.969070    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5" podUID="c42bfce2-d63b-4da3-afe0-afccb25c48cf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:33:02.969606 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:02.969189    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"
Sep 17 22:33:10.779629 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:10.779535    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 22:33:12.969224 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:12.969186    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5" podUID="c42bfce2-d63b-4da3-afe0-afccb25c48cf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:33:20.779811 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:20.779769    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 22:33:22.810562 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:22.810527    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"
Sep 17 22:33:22.879333 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:22.879257    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42bfce2-d63b-4da3-afe0-afccb25c48cf-tls-certs\") pod \"c42bfce2-d63b-4da3-afe0-afccb25c48cf\" (UID: \"c42bfce2-d63b-4da3-afe0-afccb25c48cf\") "
Sep 17 22:33:22.879495 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:22.879348    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwjhg\" (UniqueName: \"kubernetes.io/projected/c42bfce2-d63b-4da3-afe0-afccb25c48cf-kube-api-access-zwjhg\") pod \"c42bfce2-d63b-4da3-afe0-afccb25c48cf\" (UID: \"c42bfce2-d63b-4da3-afe0-afccb25c48cf\") "
Sep 17 22:33:22.881481 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:22.881450    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42bfce2-d63b-4da3-afe0-afccb25c48cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c42bfce2-d63b-4da3-afe0-afccb25c48cf" (UID: "c42bfce2-d63b-4da3-afe0-afccb25c48cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:33:22.881575 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:22.881528    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42bfce2-d63b-4da3-afe0-afccb25c48cf-kube-api-access-zwjhg" (OuterVolumeSpecName: "kube-api-access-zwjhg") pod "c42bfce2-d63b-4da3-afe0-afccb25c48cf" (UID: "c42bfce2-d63b-4da3-afe0-afccb25c48cf"). InnerVolumeSpecName "kube-api-access-zwjhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:33:22.980479 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:22.980439    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwjhg\" (UniqueName: \"kubernetes.io/projected/c42bfce2-d63b-4da3-afe0-afccb25c48cf-kube-api-access-zwjhg\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:33:22.980479 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:22.980482    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42bfce2-d63b-4da3-afe0-afccb25c48cf-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:33:23.384211 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:23.384174    2567 generic.go:356] "Generic (PLEG): container finished" podID="c42bfce2-d63b-4da3-afe0-afccb25c48cf" containerID="5614224eaf38d2cbb19fcf99f0d316f4b94dee7498d70a2c5560c6895e9bf4a4" exitCode=0
Sep 17 22:33:23.384412 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:23.384233    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"
Sep 17 22:33:23.384412 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:23.384262    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5" event={"ID":"c42bfce2-d63b-4da3-afe0-afccb25c48cf","Type":"ContainerDied","Data":"5614224eaf38d2cbb19fcf99f0d316f4b94dee7498d70a2c5560c6895e9bf4a4"}
Sep 17 22:33:23.384412 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:23.384326    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5" event={"ID":"c42bfce2-d63b-4da3-afe0-afccb25c48cf","Type":"ContainerDied","Data":"14c06770e2869d950eb4d6b80bcab9fba719e00b4e1d558ebbe969651d3a4f7d"}
Sep 17 22:33:23.384412 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:23.384358    2567 scope.go:117] "RemoveContainer" containerID="5614224eaf38d2cbb19fcf99f0d316f4b94dee7498d70a2c5560c6895e9bf4a4"
Sep 17 22:33:23.392526 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:23.392508    2567 scope.go:117] "RemoveContainer" containerID="5614224eaf38d2cbb19fcf99f0d316f4b94dee7498d70a2c5560c6895e9bf4a4"
Sep 17 22:33:23.392818 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:33:23.392800    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5614224eaf38d2cbb19fcf99f0d316f4b94dee7498d70a2c5560c6895e9bf4a4\": container with ID starting with 5614224eaf38d2cbb19fcf99f0d316f4b94dee7498d70a2c5560c6895e9bf4a4 not found: ID does not exist" containerID="5614224eaf38d2cbb19fcf99f0d316f4b94dee7498d70a2c5560c6895e9bf4a4"
Sep 17 22:33:23.392872 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:23.392828    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5614224eaf38d2cbb19fcf99f0d316f4b94dee7498d70a2c5560c6895e9bf4a4"} err="failed to get container status \"5614224eaf38d2cbb19fcf99f0d316f4b94dee7498d70a2c5560c6895e9bf4a4\": rpc error: code = NotFound desc = could not find container \"5614224eaf38d2cbb19fcf99f0d316f4b94dee7498d70a2c5560c6895e9bf4a4\": container with ID starting with 5614224eaf38d2cbb19fcf99f0d316f4b94dee7498d70a2c5560c6895e9bf4a4 not found: ID does not exist"
Sep 17 22:33:23.408324 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:23.408303    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"]
Sep 17 22:33:23.414195 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:23.414175    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schej9hl5"]
Sep 17 22:33:23.639402 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:23.639291    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c42bfce2-d63b-4da3-afe0-afccb25c48cf" path="/var/lib/kubelet/pods/c42bfce2-d63b-4da3-afe0-afccb25c48cf/volumes"
Sep 17 22:33:26.898633 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:26.898595    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"]
Sep 17 22:33:29.805460 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:29.805425    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj" podUID="3584402f-2e79-4d0c-a469-b6d7ff0f927f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:33:30.779409 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:30.779364    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 22:33:39.806058 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:39.806017    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj" podUID="3584402f-2e79-4d0c-a469-b6d7ff0f927f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:33:40.779434 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:40.779387    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 22:33:41.917260 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:41.917217    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj" podUID="3584402f-2e79-4d0c-a469-b6d7ff0f927f" containerName="main" containerID="cri-o://82e1b82eef2947ac64fd463e2358d30be7ac83f20dfeb50d22e1814e6970f6fe" gracePeriod=45
Sep 17 22:33:49.805847 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:49.805807    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj" podUID="3584402f-2e79-4d0c-a469-b6d7ff0f927f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:33:49.806310 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:49.805917    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"
Sep 17 22:33:50.778883 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:50.778840    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 22:33:59.806847 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:33:59.806808    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj" podUID="3584402f-2e79-4d0c-a469-b6d7ff0f927f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:34:00.779853 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:00.779814    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 22:34:09.805158 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:09.805109    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj" podUID="3584402f-2e79-4d0c-a469-b6d7ff0f927f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:34:10.779486 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:10.779436    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 22:34:12.153913 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.153889    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"
Sep 17 22:34:12.232637 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.232611    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2kt9\" (UniqueName: \"kubernetes.io/projected/3584402f-2e79-4d0c-a469-b6d7ff0f927f-kube-api-access-c2kt9\") pod \"3584402f-2e79-4d0c-a469-b6d7ff0f927f\" (UID: \"3584402f-2e79-4d0c-a469-b6d7ff0f927f\") "
Sep 17 22:34:12.232783 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.232698    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3584402f-2e79-4d0c-a469-b6d7ff0f927f-tls-certs\") pod \"3584402f-2e79-4d0c-a469-b6d7ff0f927f\" (UID: \"3584402f-2e79-4d0c-a469-b6d7ff0f927f\") "
Sep 17 22:34:12.234643 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.234617    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3584402f-2e79-4d0c-a469-b6d7ff0f927f-kube-api-access-c2kt9" (OuterVolumeSpecName: "kube-api-access-c2kt9") pod "3584402f-2e79-4d0c-a469-b6d7ff0f927f" (UID: "3584402f-2e79-4d0c-a469-b6d7ff0f927f"). InnerVolumeSpecName "kube-api-access-c2kt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:34:12.234758 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.234733    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3584402f-2e79-4d0c-a469-b6d7ff0f927f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3584402f-2e79-4d0c-a469-b6d7ff0f927f" (UID: "3584402f-2e79-4d0c-a469-b6d7ff0f927f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:34:12.333681 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.333650    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3584402f-2e79-4d0c-a469-b6d7ff0f927f-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:34:12.333681 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.333676    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2kt9\" (UniqueName: \"kubernetes.io/projected/3584402f-2e79-4d0c-a469-b6d7ff0f927f-kube-api-access-c2kt9\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:34:12.573416 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.573333    2567 generic.go:356] "Generic (PLEG): container finished" podID="3584402f-2e79-4d0c-a469-b6d7ff0f927f" containerID="82e1b82eef2947ac64fd463e2358d30be7ac83f20dfeb50d22e1814e6970f6fe" exitCode=0
Sep 17 22:34:12.573416 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.573374    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj" event={"ID":"3584402f-2e79-4d0c-a469-b6d7ff0f927f","Type":"ContainerDied","Data":"82e1b82eef2947ac64fd463e2358d30be7ac83f20dfeb50d22e1814e6970f6fe"}
Sep 17 22:34:12.573416 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.573411    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"
Sep 17 22:34:12.573669 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.573446    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj" event={"ID":"3584402f-2e79-4d0c-a469-b6d7ff0f927f","Type":"ContainerDied","Data":"4e5675b8ee4138487d57450704ed22f1eea43b5c753c8ef6c6c10662e96d308a"}
Sep 17 22:34:12.573669 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.573474    2567 scope.go:117] "RemoveContainer" containerID="82e1b82eef2947ac64fd463e2358d30be7ac83f20dfeb50d22e1814e6970f6fe"
Sep 17 22:34:12.581322 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.581306    2567 scope.go:117] "RemoveContainer" containerID="82e1b82eef2947ac64fd463e2358d30be7ac83f20dfeb50d22e1814e6970f6fe"
Sep 17 22:34:12.581568 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:34:12.581553    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e1b82eef2947ac64fd463e2358d30be7ac83f20dfeb50d22e1814e6970f6fe\": container with ID starting with 82e1b82eef2947ac64fd463e2358d30be7ac83f20dfeb50d22e1814e6970f6fe not found: ID does not exist" containerID="82e1b82eef2947ac64fd463e2358d30be7ac83f20dfeb50d22e1814e6970f6fe"
Sep 17 22:34:12.581623 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.581575    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e1b82eef2947ac64fd463e2358d30be7ac83f20dfeb50d22e1814e6970f6fe"} err="failed to get container status \"82e1b82eef2947ac64fd463e2358d30be7ac83f20dfeb50d22e1814e6970f6fe\": rpc error: code = NotFound desc = could not find container \"82e1b82eef2947ac64fd463e2358d30be7ac83f20dfeb50d22e1814e6970f6fe\": container with ID starting with 82e1b82eef2947ac64fd463e2358d30be7ac83f20dfeb50d22e1814e6970f6fe not found: ID does not exist"
Sep 17 22:34:12.597098 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.597077    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"]
Sep 17 22:34:12.599748 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:12.599727    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegg6qj"]
Sep 17 22:34:13.640493 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:13.640454    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3584402f-2e79-4d0c-a469-b6d7ff0f927f" path="/var/lib/kubelet/pods/3584402f-2e79-4d0c-a469-b6d7ff0f927f/volumes"
Sep 17 22:34:20.779662 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:20.779615    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 22:34:30.789342 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:30.789283    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:34:30.797375 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:30.797351    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:34:31.400147 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.400116    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"]
Sep 17 22:34:31.400387 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.400376    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3584402f-2e79-4d0c-a469-b6d7ff0f927f" containerName="main"
Sep 17 22:34:31.400440 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.400389    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3584402f-2e79-4d0c-a469-b6d7ff0f927f" containerName="main"
Sep 17 22:34:31.400440 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.400400    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42bfce2-d63b-4da3-afe0-afccb25c48cf" containerName="main"
Sep 17 22:34:31.400440 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.400405    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42bfce2-d63b-4da3-afe0-afccb25c48cf" containerName="main"
Sep 17 22:34:31.400543 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.400456    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42bfce2-d63b-4da3-afe0-afccb25c48cf" containerName="main"
Sep 17 22:34:31.400543 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.400464    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3584402f-2e79-4d0c-a469-b6d7ff0f927f" containerName="main"
Sep 17 22:34:31.404582 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.404564    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"
Sep 17 22:34:31.407575 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.407553    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 22:34:31.408050 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.408032    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 22:34:31.408126 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.408067    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 22:34:31.408437 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.408420    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-bt8z8\""
Sep 17 22:34:31.408517 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.408454    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-vsbj6\""
Sep 17 22:34:31.415776 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.415753    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"]
Sep 17 22:34:31.465993 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.465969    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6663990-283c-4d47-88d0-9d2150997cd2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4\" (UID: \"e6663990-283c-4d47-88d0-9d2150997cd2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"
Sep 17 22:34:31.466142 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.466025    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q79s\" (UniqueName: \"kubernetes.io/projected/e6663990-283c-4d47-88d0-9d2150997cd2-kube-api-access-7q79s\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4\" (UID: \"e6663990-283c-4d47-88d0-9d2150997cd2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"
Sep 17 22:34:31.567307 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.567283    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6663990-283c-4d47-88d0-9d2150997cd2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4\" (UID: \"e6663990-283c-4d47-88d0-9d2150997cd2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"
Sep 17 22:34:31.567412 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.567323    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q79s\" (UniqueName: \"kubernetes.io/projected/e6663990-283c-4d47-88d0-9d2150997cd2-kube-api-access-7q79s\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4\" (UID: \"e6663990-283c-4d47-88d0-9d2150997cd2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"
Sep 17 22:34:31.569632 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.569612    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6663990-283c-4d47-88d0-9d2150997cd2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4\" (UID: \"e6663990-283c-4d47-88d0-9d2150997cd2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"
Sep 17 22:34:31.575220 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.575197    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q79s\" (UniqueName: \"kubernetes.io/projected/e6663990-283c-4d47-88d0-9d2150997cd2-kube-api-access-7q79s\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4\" (UID: \"e6663990-283c-4d47-88d0-9d2150997cd2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"
Sep 17 22:34:31.714043 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.713965    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"
Sep 17 22:34:31.988579 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:31.988554    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"]
Sep 17 22:34:31.991066 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:34:31.991035    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6663990_283c_4d47_88d0_9d2150997cd2.slice/crio-183c5fded51e3f1a796e026bde743d3634c4c3f0e27c6cc0e55c986158f8a92d WatchSource:0}: Error finding container 183c5fded51e3f1a796e026bde743d3634c4c3f0e27c6cc0e55c986158f8a92d: Status 404 returned error can't find the container with id 183c5fded51e3f1a796e026bde743d3634c4c3f0e27c6cc0e55c986158f8a92d
Sep 17 22:34:32.650902 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:32.650863    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4" event={"ID":"e6663990-283c-4d47-88d0-9d2150997cd2","Type":"ContainerStarted","Data":"d3a20f50995b92b0be0e9942f9059334018f7e76de71bc75bdc592ae56825d71"}
Sep 17 22:34:32.651102 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:32.650993    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"
Sep 17 22:34:32.651102 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:32.651050    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4" event={"ID":"e6663990-283c-4d47-88d0-9d2150997cd2","Type":"ContainerStarted","Data":"183c5fded51e3f1a796e026bde743d3634c4c3f0e27c6cc0e55c986158f8a92d"}
Sep 17 22:34:32.666880 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:32.666837    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4" podStartSLOduration=1.666822966 podStartE2EDuration="1.666822966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:34:32.665874827 +0000 UTC m=+2149.674804380" watchObservedRunningTime="2026-09-17 22:34:32.666822966 +0000 UTC m=+2149.675752513"
Sep 17 22:34:56.894450 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:34:56.894417    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"]
Sep 17 22:35:03.656140 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:03.656108    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"
Sep 17 22:35:11.916863 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:11.916818    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="main" containerID="cri-o://6226ec5798561285a217577be43765c9265626e4830e3b8ec62237b2920a6d7a" gracePeriod=45
Sep 17 22:35:12.662625 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.662602    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:35:12.762925 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.762895    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-dshm\") pod \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") "
Sep 17 22:35:12.763092 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.762944    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f5997f-c74a-4fef-b3d5-4a6677576f1c-tls-certs\") pod \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") "
Sep 17 22:35:12.763092 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.762995    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-tmp-dir\") pod \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") "
Sep 17 22:35:12.763092 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.763044    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-model-cache\") pod \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") "
Sep 17 22:35:12.763092 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.763072    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38f5997f-c74a-4fef-b3d5-4a6677576f1c-cabundle-cert\") pod \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") "
Sep 17 22:35:12.763292 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.763112    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-home\") pod \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") "
Sep 17 22:35:12.763292 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.763145    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh48f\" (UniqueName: \"kubernetes.io/projected/38f5997f-c74a-4fef-b3d5-4a6677576f1c-kube-api-access-rh48f\") pod \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") "
Sep 17 22:35:12.763292 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.763183    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-kserve-provision-location\") pod \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\" (UID: \"38f5997f-c74a-4fef-b3d5-4a6677576f1c\") "
Sep 17 22:35:12.763443 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.763356    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-model-cache" (OuterVolumeSpecName: "model-cache") pod "38f5997f-c74a-4fef-b3d5-4a6677576f1c" (UID: "38f5997f-c74a-4fef-b3d5-4a6677576f1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:35:12.763443 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.763438    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:35:12.763747 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.763719    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f5997f-c74a-4fef-b3d5-4a6677576f1c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "38f5997f-c74a-4fef-b3d5-4a6677576f1c" (UID: "38f5997f-c74a-4fef-b3d5-4a6677576f1c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:35:12.763869 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.763844    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-home" (OuterVolumeSpecName: "home") pod "38f5997f-c74a-4fef-b3d5-4a6677576f1c" (UID: "38f5997f-c74a-4fef-b3d5-4a6677576f1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:35:12.765605 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.765575    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-dshm" (OuterVolumeSpecName: "dshm") pod "38f5997f-c74a-4fef-b3d5-4a6677576f1c" (UID: "38f5997f-c74a-4fef-b3d5-4a6677576f1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:35:12.765701 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.765602    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f5997f-c74a-4fef-b3d5-4a6677576f1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38f5997f-c74a-4fef-b3d5-4a6677576f1c" (UID: "38f5997f-c74a-4fef-b3d5-4a6677576f1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:35:12.765701 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.765611    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f5997f-c74a-4fef-b3d5-4a6677576f1c-kube-api-access-rh48f" (OuterVolumeSpecName: "kube-api-access-rh48f") pod "38f5997f-c74a-4fef-b3d5-4a6677576f1c" (UID: "38f5997f-c74a-4fef-b3d5-4a6677576f1c"). InnerVolumeSpecName "kube-api-access-rh48f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:35:12.774826 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.774800    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38f5997f-c74a-4fef-b3d5-4a6677576f1c" (UID: "38f5997f-c74a-4fef-b3d5-4a6677576f1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:35:12.783367 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.783346    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38f5997f-c74a-4fef-b3d5-4a6677576f1c" (UID: "38f5997f-c74a-4fef-b3d5-4a6677576f1c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:35:12.796280 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.796253    2567 generic.go:356] "Generic (PLEG): container finished" podID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerID="6226ec5798561285a217577be43765c9265626e4830e3b8ec62237b2920a6d7a" exitCode=0
Sep 17 22:35:12.796364 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.796303    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"
Sep 17 22:35:12.796364 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.796335    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" event={"ID":"38f5997f-c74a-4fef-b3d5-4a6677576f1c","Type":"ContainerDied","Data":"6226ec5798561285a217577be43765c9265626e4830e3b8ec62237b2920a6d7a"}
Sep 17 22:35:12.796434 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.796390    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl" event={"ID":"38f5997f-c74a-4fef-b3d5-4a6677576f1c","Type":"ContainerDied","Data":"e89f7d3acca9ddb5cc52531f3e6a815132ba5497e0a41ddcf95010adf5484e7a"}
Sep 17 22:35:12.796434 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.796421    2567 scope.go:117] "RemoveContainer" containerID="6226ec5798561285a217577be43765c9265626e4830e3b8ec62237b2920a6d7a"
Sep 17 22:35:12.811691 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.811633    2567 scope.go:117] "RemoveContainer" containerID="b18b769991ff33d55f34c6af39cbf68717b0ea673baff7c5bd08b29d81754046"
Sep 17 22:35:12.818923 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.818908    2567 scope.go:117] "RemoveContainer" containerID="6226ec5798561285a217577be43765c9265626e4830e3b8ec62237b2920a6d7a"
Sep 17 22:35:12.819197 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:35:12.819180    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6226ec5798561285a217577be43765c9265626e4830e3b8ec62237b2920a6d7a\": container with ID starting with 6226ec5798561285a217577be43765c9265626e4830e3b8ec62237b2920a6d7a not found: ID does not exist" containerID="6226ec5798561285a217577be43765c9265626e4830e3b8ec62237b2920a6d7a"
Sep 17 22:35:12.819263 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.819205    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6226ec5798561285a217577be43765c9265626e4830e3b8ec62237b2920a6d7a"} err="failed to get container status \"6226ec5798561285a217577be43765c9265626e4830e3b8ec62237b2920a6d7a\": rpc error: code = NotFound desc = could not find container \"6226ec5798561285a217577be43765c9265626e4830e3b8ec62237b2920a6d7a\": container with ID starting with 6226ec5798561285a217577be43765c9265626e4830e3b8ec62237b2920a6d7a not found: ID does not exist"
Sep 17 22:35:12.819263 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.819225    2567 scope.go:117] "RemoveContainer" containerID="b18b769991ff33d55f34c6af39cbf68717b0ea673baff7c5bd08b29d81754046"
Sep 17 22:35:12.819501 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:35:12.819483    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18b769991ff33d55f34c6af39cbf68717b0ea673baff7c5bd08b29d81754046\": container with ID starting with b18b769991ff33d55f34c6af39cbf68717b0ea673baff7c5bd08b29d81754046 not found: ID does not exist" containerID="b18b769991ff33d55f34c6af39cbf68717b0ea673baff7c5bd08b29d81754046"
Sep 17 22:35:12.819549 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.819506    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18b769991ff33d55f34c6af39cbf68717b0ea673baff7c5bd08b29d81754046"} err="failed to get container status \"b18b769991ff33d55f34c6af39cbf68717b0ea673baff7c5bd08b29d81754046\": rpc error: code = NotFound desc = could not find container \"b18b769991ff33d55f34c6af39cbf68717b0ea673baff7c5bd08b29d81754046\": container with ID starting with b18b769991ff33d55f34c6af39cbf68717b0ea673baff7c5bd08b29d81754046 not found: ID does not exist"
Sep 17 22:35:12.822096 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.822075    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"]
Sep 17 22:35:12.825370 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.825350    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-95df8d9c-7wbjl"]
Sep 17 22:35:12.864190 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.864159    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:35:12.864190 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.864185    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:35:12.864190 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.864195    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f5997f-c74a-4fef-b3d5-4a6677576f1c-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:35:12.864379 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.864224    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:35:12.864379 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.864233    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38f5997f-c74a-4fef-b3d5-4a6677576f1c-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:35:12.864379 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.864240    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f5997f-c74a-4fef-b3d5-4a6677576f1c-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:35:12.864379 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:12.864249    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh48f\" (UniqueName: \"kubernetes.io/projected/38f5997f-c74a-4fef-b3d5-4a6677576f1c-kube-api-access-rh48f\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:35:13.639636 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:35:13.639601    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" path="/var/lib/kubelet/pods/38f5997f-c74a-4fef-b3d5-4a6677576f1c/volumes"
Sep 17 22:36:03.692369 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.692325    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"]
Sep 17 22:36:03.693165 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.692579    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="storage-initializer"
Sep 17 22:36:03.693165 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.692590    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="storage-initializer"
Sep 17 22:36:03.693165 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.692613    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="main"
Sep 17 22:36:03.693165 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.692619    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="main"
Sep 17 22:36:03.693165 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.692657    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="38f5997f-c74a-4fef-b3d5-4a6677576f1c" containerName="main"
Sep 17 22:36:03.695418 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.695394    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"
Sep 17 22:36:03.698879 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.698853    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 22:36:03.698991 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.698885    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 22:36:03.698991 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.698855    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-mwztz\""
Sep 17 22:36:03.698991 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.698857    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-zwlz6\""
Sep 17 22:36:03.698991 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.698864    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 22:36:03.704441 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.704419    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"]
Sep 17 22:36:03.714527 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.714507    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfe64d1-c966-4a9f-b22d-668da6ea86a5-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b\" (UID: \"0cfe64d1-c966-4a9f-b22d-668da6ea86a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"
Sep 17 22:36:03.714616 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.714537    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w4qw\" (UniqueName: \"kubernetes.io/projected/0cfe64d1-c966-4a9f-b22d-668da6ea86a5-kube-api-access-6w4qw\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b\" (UID: \"0cfe64d1-c966-4a9f-b22d-668da6ea86a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"
Sep 17 22:36:03.815049 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.815022    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfe64d1-c966-4a9f-b22d-668da6ea86a5-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b\" (UID: \"0cfe64d1-c966-4a9f-b22d-668da6ea86a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"
Sep 17 22:36:03.815212 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.815058    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w4qw\" (UniqueName: \"kubernetes.io/projected/0cfe64d1-c966-4a9f-b22d-668da6ea86a5-kube-api-access-6w4qw\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b\" (UID: \"0cfe64d1-c966-4a9f-b22d-668da6ea86a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"
Sep 17 22:36:03.817437 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.817418    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfe64d1-c966-4a9f-b22d-668da6ea86a5-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b\" (UID: \"0cfe64d1-c966-4a9f-b22d-668da6ea86a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"
Sep 17 22:36:03.822826 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:03.822793    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w4qw\" (UniqueName: \"kubernetes.io/projected/0cfe64d1-c966-4a9f-b22d-668da6ea86a5-kube-api-access-6w4qw\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b\" (UID: \"0cfe64d1-c966-4a9f-b22d-668da6ea86a5\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"
Sep 17 22:36:04.007601 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:04.007560    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"
Sep 17 22:36:04.298251 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:04.298172    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"]
Sep 17 22:36:04.301506 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:36:04.301477    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cfe64d1_c966_4a9f_b22d_668da6ea86a5.slice/crio-70b3c097d67581958f016b82d832f80ee7b4735debfaf76d5afcd6a8085a1e46 WatchSource:0}: Error finding container 70b3c097d67581958f016b82d832f80ee7b4735debfaf76d5afcd6a8085a1e46: Status 404 returned error can't find the container with id 70b3c097d67581958f016b82d832f80ee7b4735debfaf76d5afcd6a8085a1e46
Sep 17 22:36:04.977899 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:04.977864    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b" event={"ID":"0cfe64d1-c966-4a9f-b22d-668da6ea86a5","Type":"ContainerStarted","Data":"d88c89fe2a04e02dbca87b2f98bd749315eeef2b32b0cc13b41525f5a9d2be72"}
Sep 17 22:36:04.978297 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:04.977923    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b" event={"ID":"0cfe64d1-c966-4a9f-b22d-668da6ea86a5","Type":"ContainerStarted","Data":"70b3c097d67581958f016b82d832f80ee7b4735debfaf76d5afcd6a8085a1e46"}
Sep 17 22:36:04.978297 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:04.978018    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"
Sep 17 22:36:04.994900 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:04.994853    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b" podStartSLOduration=1.994839941 podStartE2EDuration="1.994839941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:36:04.993819351 +0000 UTC m=+2242.002748894" watchObservedRunningTime="2026-09-17 22:36:04.994839941 +0000 UTC m=+2242.003769495"
Sep 17 22:36:35.984245 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:36:35.984169    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"
Sep 17 22:37:10.475042 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:10.474987    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"]
Sep 17 22:37:13.656323 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:13.656287    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4" podUID="e6663990-283c-4d47-88d0-9d2150997cd2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:37:23.656450 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:23.656407    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4" podUID="e6663990-283c-4d47-88d0-9d2150997cd2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:37:25.496144 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:25.496108    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4" podUID="e6663990-283c-4d47-88d0-9d2150997cd2" containerName="main" containerID="cri-o://d3a20f50995b92b0be0e9942f9059334018f7e76de71bc75bdc592ae56825d71" gracePeriod=45
Sep 17 22:37:33.656475 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:33.656439    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4" podUID="e6663990-283c-4d47-88d0-9d2150997cd2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:37:33.657011 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:33.656540    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"
Sep 17 22:37:43.655727 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:43.655694    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4" podUID="e6663990-283c-4d47-88d0-9d2150997cd2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:37:53.655687 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:53.655650    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4" podUID="e6663990-283c-4d47-88d0-9d2150997cd2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:37:55.746806 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:55.746783    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"
Sep 17 22:37:55.797542 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:55.797483    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q79s\" (UniqueName: \"kubernetes.io/projected/e6663990-283c-4d47-88d0-9d2150997cd2-kube-api-access-7q79s\") pod \"e6663990-283c-4d47-88d0-9d2150997cd2\" (UID: \"e6663990-283c-4d47-88d0-9d2150997cd2\") "
Sep 17 22:37:55.797674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:55.797552    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6663990-283c-4d47-88d0-9d2150997cd2-tls-certs\") pod \"e6663990-283c-4d47-88d0-9d2150997cd2\" (UID: \"e6663990-283c-4d47-88d0-9d2150997cd2\") "
Sep 17 22:37:55.799674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:55.799650    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6663990-283c-4d47-88d0-9d2150997cd2-kube-api-access-7q79s" (OuterVolumeSpecName: "kube-api-access-7q79s") pod "e6663990-283c-4d47-88d0-9d2150997cd2" (UID: "e6663990-283c-4d47-88d0-9d2150997cd2"). InnerVolumeSpecName "kube-api-access-7q79s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:37:55.799760 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:55.799702    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6663990-283c-4d47-88d0-9d2150997cd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6663990-283c-4d47-88d0-9d2150997cd2" (UID: "e6663990-283c-4d47-88d0-9d2150997cd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:37:55.898497 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:55.898474    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6663990-283c-4d47-88d0-9d2150997cd2-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:37:55.898497 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:55.898495    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q79s\" (UniqueName: \"kubernetes.io/projected/e6663990-283c-4d47-88d0-9d2150997cd2-kube-api-access-7q79s\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:37:56.373799 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:56.373765    2567 generic.go:356] "Generic (PLEG): container finished" podID="e6663990-283c-4d47-88d0-9d2150997cd2" containerID="d3a20f50995b92b0be0e9942f9059334018f7e76de71bc75bdc592ae56825d71" exitCode=0
Sep 17 22:37:56.373947 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:56.373811    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"
Sep 17 22:37:56.373947 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:56.373813    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4" event={"ID":"e6663990-283c-4d47-88d0-9d2150997cd2","Type":"ContainerDied","Data":"d3a20f50995b92b0be0e9942f9059334018f7e76de71bc75bdc592ae56825d71"}
Sep 17 22:37:56.373947 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:56.373932    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4" event={"ID":"e6663990-283c-4d47-88d0-9d2150997cd2","Type":"ContainerDied","Data":"183c5fded51e3f1a796e026bde743d3634c4c3f0e27c6cc0e55c986158f8a92d"}
Sep 17 22:37:56.374131 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:56.373984    2567 scope.go:117] "RemoveContainer" containerID="d3a20f50995b92b0be0e9942f9059334018f7e76de71bc75bdc592ae56825d71"
Sep 17 22:37:56.382387 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:56.382373    2567 scope.go:117] "RemoveContainer" containerID="d3a20f50995b92b0be0e9942f9059334018f7e76de71bc75bdc592ae56825d71"
Sep 17 22:37:56.382641 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:37:56.382624    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3a20f50995b92b0be0e9942f9059334018f7e76de71bc75bdc592ae56825d71\": container with ID starting with d3a20f50995b92b0be0e9942f9059334018f7e76de71bc75bdc592ae56825d71 not found: ID does not exist" containerID="d3a20f50995b92b0be0e9942f9059334018f7e76de71bc75bdc592ae56825d71"
Sep 17 22:37:56.382685 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:56.382648    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a20f50995b92b0be0e9942f9059334018f7e76de71bc75bdc592ae56825d71"} err="failed to get container status \"d3a20f50995b92b0be0e9942f9059334018f7e76de71bc75bdc592ae56825d71\": rpc error: code = NotFound desc = could not find container \"d3a20f50995b92b0be0e9942f9059334018f7e76de71bc75bdc592ae56825d71\": container with ID starting with d3a20f50995b92b0be0e9942f9059334018f7e76de71bc75bdc592ae56825d71 not found: ID does not exist"
Sep 17 22:37:56.394223 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:56.394197    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"]
Sep 17 22:37:56.397223 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:56.397202    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheqzxl4"]
Sep 17 22:37:57.639155 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:37:57.639120    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6663990-283c-4d47-88d0-9d2150997cd2" path="/var/lib/kubelet/pods/e6663990-283c-4d47-88d0-9d2150997cd2/volumes"
Sep 17 22:38:07.847703 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.847668    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"]
Sep 17 22:38:07.848162 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.847937    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6663990-283c-4d47-88d0-9d2150997cd2" containerName="main"
Sep 17 22:38:07.848162 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.847948    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6663990-283c-4d47-88d0-9d2150997cd2" containerName="main"
Sep 17 22:38:07.848162 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.848023    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6663990-283c-4d47-88d0-9d2150997cd2" containerName="main"
Sep 17 22:38:07.852325 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.852303    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:07.854876 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.854857    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 22:38:07.855092 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.854859    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 22:38:07.855092 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.855029    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-v6sqj\""
Sep 17 22:38:07.855092 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.855041    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 22:38:07.855296 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.855281    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 17 22:38:07.855511 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.855498    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:38:07.863362 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.863336    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"]
Sep 17 22:38:07.985267 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.985244    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:07.985384 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.985275    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e46743-e695-4a88-9a66-7d724622d04f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:07.985384 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.985298    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:07.985384 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.985354    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:07.985514 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.985389    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:07.985514 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.985406    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxwks\" (UniqueName: \"kubernetes.io/projected/f8e46743-e695-4a88-9a66-7d724622d04f-kube-api-access-nxwks\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:07.985514 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.985446    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:07.985514 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:07.985463    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e46743-e695-4a88-9a66-7d724622d04f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.086625 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.086603    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.086752 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.086638    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.086752 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.086666    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.086752 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.086688    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxwks\" (UniqueName: \"kubernetes.io/projected/f8e46743-e695-4a88-9a66-7d724622d04f-kube-api-access-nxwks\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.086917 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.086753    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.086917 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.086780    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e46743-e695-4a88-9a66-7d724622d04f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.086917 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.086816    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.086917 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.086847    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e46743-e695-4a88-9a66-7d724622d04f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.087159 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.087098    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.087290 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.087267    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.087447 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.087337    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.087549 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.087481    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e46743-e695-4a88-9a66-7d724622d04f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.087643 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.087487    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.089523 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.089504    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.089972 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.089951    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e46743-e695-4a88-9a66-7d724622d04f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.099988 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.099934    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxwks\" (UniqueName: \"kubernetes.io/projected/f8e46743-e695-4a88-9a66-7d724622d04f-kube-api-access-nxwks\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.162859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.162833    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:08.440159 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:08.439984    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"]
Sep 17 22:38:08.442124 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:38:08.442099    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e46743_e695_4a88_9a66_7d724622d04f.slice/crio-0515e5fe54f7339253f209b00333bd1a96783b4f6d595c29441d0c314608300f WatchSource:0}: Error finding container 0515e5fe54f7339253f209b00333bd1a96783b4f6d595c29441d0c314608300f: Status 404 returned error can't find the container with id 0515e5fe54f7339253f209b00333bd1a96783b4f6d595c29441d0c314608300f
Sep 17 22:38:09.423355 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:09.423321    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" event={"ID":"f8e46743-e695-4a88-9a66-7d724622d04f","Type":"ContainerStarted","Data":"e8e6317aafa537d225ed90e85b95a5c8ef6c8f81b39d93be63af5b45a9070b3d"}
Sep 17 22:38:09.423721 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:09.423375    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" event={"ID":"f8e46743-e695-4a88-9a66-7d724622d04f","Type":"ContainerStarted","Data":"0515e5fe54f7339253f209b00333bd1a96783b4f6d595c29441d0c314608300f"}
Sep 17 22:38:11.432270 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:11.432184    2567 generic.go:356] "Generic (PLEG): container finished" podID="f8e46743-e695-4a88-9a66-7d724622d04f" containerID="e8e6317aafa537d225ed90e85b95a5c8ef6c8f81b39d93be63af5b45a9070b3d" exitCode=0
Sep 17 22:38:11.432619 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:11.432256    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" event={"ID":"f8e46743-e695-4a88-9a66-7d724622d04f","Type":"ContainerDied","Data":"e8e6317aafa537d225ed90e85b95a5c8ef6c8f81b39d93be63af5b45a9070b3d"}
Sep 17 22:38:12.437686 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:12.437653    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" event={"ID":"f8e46743-e695-4a88-9a66-7d724622d04f","Type":"ContainerStarted","Data":"85359b0d39a8d438519949471604788827a9751add9fd7573d48ab78dfd5bf0e"}
Sep 17 22:38:12.463795 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:12.463741    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" podStartSLOduration=2.889678505 podStartE2EDuration="5.463725637s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.574047132s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:38:12.462742805 +0000 UTC m=+2369.471672347" watchObservedRunningTime="2026-09-17 22:38:12.463725637 +0000 UTC m=+2369.472655178"
Sep 17 22:38:13.163625 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:13.163589    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:18.163592 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:18.163545    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:38:18.165017 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:18.164979    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 22:38:23.799627 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:23.799574    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"]
Sep 17 22:38:25.984811 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:25.984768    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b" podUID="0cfe64d1-c966-4a9f-b22d-668da6ea86a5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:38:28.163985 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:28.163939    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 22:38:35.984037 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:35.983979    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b" podUID="0cfe64d1-c966-4a9f-b22d-668da6ea86a5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:38:38.164080 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:38.164036    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 22:38:38.818441 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:38.818384    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b" podUID="0cfe64d1-c966-4a9f-b22d-668da6ea86a5" containerName="main" containerID="cri-o://d88c89fe2a04e02dbca87b2f98bd749315eeef2b32b0cc13b41525f5a9d2be72" gracePeriod=45
Sep 17 22:38:45.984831 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:45.984793    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b" podUID="0cfe64d1-c966-4a9f-b22d-668da6ea86a5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:38:45.985352 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:45.984903    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"
Sep 17 22:38:48.164269 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:48.164226    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 22:38:55.984137 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:55.984091    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b" podUID="0cfe64d1-c966-4a9f-b22d-668da6ea86a5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:38:58.163886 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:38:58.163837    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 22:39:05.984583 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:05.984503    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b" podUID="0cfe64d1-c966-4a9f-b22d-668da6ea86a5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:39:08.163955 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:08.163907    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 22:39:09.063938 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.063910    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"
Sep 17 22:39:09.157159 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.157065    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfe64d1-c966-4a9f-b22d-668da6ea86a5-tls-certs\") pod \"0cfe64d1-c966-4a9f-b22d-668da6ea86a5\" (UID: \"0cfe64d1-c966-4a9f-b22d-668da6ea86a5\") "
Sep 17 22:39:09.157159 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.157131    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w4qw\" (UniqueName: \"kubernetes.io/projected/0cfe64d1-c966-4a9f-b22d-668da6ea86a5-kube-api-access-6w4qw\") pod \"0cfe64d1-c966-4a9f-b22d-668da6ea86a5\" (UID: \"0cfe64d1-c966-4a9f-b22d-668da6ea86a5\") "
Sep 17 22:39:09.159286 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.159257    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cfe64d1-c966-4a9f-b22d-668da6ea86a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0cfe64d1-c966-4a9f-b22d-668da6ea86a5" (UID: "0cfe64d1-c966-4a9f-b22d-668da6ea86a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:39:09.159416 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.159401    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cfe64d1-c966-4a9f-b22d-668da6ea86a5-kube-api-access-6w4qw" (OuterVolumeSpecName: "kube-api-access-6w4qw") pod "0cfe64d1-c966-4a9f-b22d-668da6ea86a5" (UID: "0cfe64d1-c966-4a9f-b22d-668da6ea86a5"). InnerVolumeSpecName "kube-api-access-6w4qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:39:09.257701 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.257670    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w4qw\" (UniqueName: \"kubernetes.io/projected/0cfe64d1-c966-4a9f-b22d-668da6ea86a5-kube-api-access-6w4qw\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:39:09.257701 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.257696    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfe64d1-c966-4a9f-b22d-668da6ea86a5-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:39:09.659282 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.659252    2567 generic.go:356] "Generic (PLEG): container finished" podID="0cfe64d1-c966-4a9f-b22d-668da6ea86a5" containerID="d88c89fe2a04e02dbca87b2f98bd749315eeef2b32b0cc13b41525f5a9d2be72" exitCode=0
Sep 17 22:39:09.659465 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.659290    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"
Sep 17 22:39:09.659465 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.659295    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b" event={"ID":"0cfe64d1-c966-4a9f-b22d-668da6ea86a5","Type":"ContainerDied","Data":"d88c89fe2a04e02dbca87b2f98bd749315eeef2b32b0cc13b41525f5a9d2be72"}
Sep 17 22:39:09.659465 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.659343    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b" event={"ID":"0cfe64d1-c966-4a9f-b22d-668da6ea86a5","Type":"ContainerDied","Data":"70b3c097d67581958f016b82d832f80ee7b4735debfaf76d5afcd6a8085a1e46"}
Sep 17 22:39:09.659465 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.659389    2567 scope.go:117] "RemoveContainer" containerID="d88c89fe2a04e02dbca87b2f98bd749315eeef2b32b0cc13b41525f5a9d2be72"
Sep 17 22:39:09.669585 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.669561    2567 scope.go:117] "RemoveContainer" containerID="d88c89fe2a04e02dbca87b2f98bd749315eeef2b32b0cc13b41525f5a9d2be72"
Sep 17 22:39:09.670049 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:39:09.670018    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88c89fe2a04e02dbca87b2f98bd749315eeef2b32b0cc13b41525f5a9d2be72\": container with ID starting with d88c89fe2a04e02dbca87b2f98bd749315eeef2b32b0cc13b41525f5a9d2be72 not found: ID does not exist" containerID="d88c89fe2a04e02dbca87b2f98bd749315eeef2b32b0cc13b41525f5a9d2be72"
Sep 17 22:39:09.670156 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.670054    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88c89fe2a04e02dbca87b2f98bd749315eeef2b32b0cc13b41525f5a9d2be72"} err="failed to get container status \"d88c89fe2a04e02dbca87b2f98bd749315eeef2b32b0cc13b41525f5a9d2be72\": rpc error: code = NotFound desc = could not find container \"d88c89fe2a04e02dbca87b2f98bd749315eeef2b32b0cc13b41525f5a9d2be72\": container with ID starting with d88c89fe2a04e02dbca87b2f98bd749315eeef2b32b0cc13b41525f5a9d2be72 not found: ID does not exist"
Sep 17 22:39:09.675944 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.675916    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"]
Sep 17 22:39:09.678299 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:09.678276    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schelk47b"]
Sep 17 22:39:11.640226 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:11.640188    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cfe64d1-c966-4a9f-b22d-668da6ea86a5" path="/var/lib/kubelet/pods/0cfe64d1-c966-4a9f-b22d-668da6ea86a5/volumes"
Sep 17 22:39:18.163534 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:18.163492    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 22:39:19.749242 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.749203    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"]
Sep 17 22:39:19.749745 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.749593    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cfe64d1-c966-4a9f-b22d-668da6ea86a5" containerName="main"
Sep 17 22:39:19.749745 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.749612    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cfe64d1-c966-4a9f-b22d-668da6ea86a5" containerName="main"
Sep 17 22:39:19.749745 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.749684    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cfe64d1-c966-4a9f-b22d-668da6ea86a5" containerName="main"
Sep 17 22:39:19.753956 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.753933    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"
Sep 17 22:39:19.757265 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.757232    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-2c7f8\""
Sep 17 22:39:19.757538 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.757519    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 22:39:19.757870 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.757851    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 22:39:19.758212 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.758194    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-bdswc\""
Sep 17 22:39:19.758602 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.758585    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 22:39:19.772013 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.771972    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"]
Sep 17 22:39:19.855813 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.855780    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c28debd-4c15-4fa3-b4e4-e7b9298df064-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2\" (UID: \"3c28debd-4c15-4fa3-b4e4-e7b9298df064\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"
Sep 17 22:39:19.855942 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.855826    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r5lj\" (UniqueName: \"kubernetes.io/projected/3c28debd-4c15-4fa3-b4e4-e7b9298df064-kube-api-access-9r5lj\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2\" (UID: \"3c28debd-4c15-4fa3-b4e4-e7b9298df064\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"
Sep 17 22:39:19.957317 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.957281    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c28debd-4c15-4fa3-b4e4-e7b9298df064-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2\" (UID: \"3c28debd-4c15-4fa3-b4e4-e7b9298df064\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"
Sep 17 22:39:19.957453 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.957337    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r5lj\" (UniqueName: \"kubernetes.io/projected/3c28debd-4c15-4fa3-b4e4-e7b9298df064-kube-api-access-9r5lj\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2\" (UID: \"3c28debd-4c15-4fa3-b4e4-e7b9298df064\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"
Sep 17 22:39:19.960499 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.960470    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c28debd-4c15-4fa3-b4e4-e7b9298df064-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2\" (UID: \"3c28debd-4c15-4fa3-b4e4-e7b9298df064\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"
Sep 17 22:39:19.966656 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:19.966630    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r5lj\" (UniqueName: \"kubernetes.io/projected/3c28debd-4c15-4fa3-b4e4-e7b9298df064-kube-api-access-9r5lj\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2\" (UID: \"3c28debd-4c15-4fa3-b4e4-e7b9298df064\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"
Sep 17 22:39:20.069902 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:20.069814    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"
Sep 17 22:39:20.345373 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:20.345285    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"]
Sep 17 22:39:20.349378 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:39:20.349330    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c28debd_4c15_4fa3_b4e4_e7b9298df064.slice/crio-d932d9edc72843d0943274bf9d8b89a17b3cbc7d42d8845bff2faebd88ec06f8 WatchSource:0}: Error finding container d932d9edc72843d0943274bf9d8b89a17b3cbc7d42d8845bff2faebd88ec06f8: Status 404 returned error can't find the container with id d932d9edc72843d0943274bf9d8b89a17b3cbc7d42d8845bff2faebd88ec06f8
Sep 17 22:39:20.703439 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:20.703336    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2" event={"ID":"3c28debd-4c15-4fa3-b4e4-e7b9298df064","Type":"ContainerStarted","Data":"91ae6b7e8fae2c36b45929e4ae11bb6c1e1559078fae79c293cd848c3c448550"}
Sep 17 22:39:20.703439 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:20.703389    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2" event={"ID":"3c28debd-4c15-4fa3-b4e4-e7b9298df064","Type":"ContainerStarted","Data":"d932d9edc72843d0943274bf9d8b89a17b3cbc7d42d8845bff2faebd88ec06f8"}
Sep 17 22:39:20.703439 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:20.703425    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"
Sep 17 22:39:20.719065 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:20.719016    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2" podStartSLOduration=1.7189856799999998 podStartE2EDuration="1.71898568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:39:20.718874576 +0000 UTC m=+2437.727804116" watchObservedRunningTime="2026-09-17 22:39:20.71898568 +0000 UTC m=+2437.727915220"
Sep 17 22:39:28.173514 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:28.173481    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:39:28.180873 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:28.180851    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:39:34.032051 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:34.032012    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"]
Sep 17 22:39:49.053957 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.053899    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" containerName="main" containerID="cri-o://85359b0d39a8d438519949471604788827a9751add9fd7573d48ab78dfd5bf0e" gracePeriod=45
Sep 17 22:39:49.174950 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.174908    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 17 22:39:49.688478 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.688455    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:39:49.773380 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773314    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-tmp-dir\") pod \"f8e46743-e695-4a88-9a66-7d724622d04f\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") "
Sep 17 22:39:49.773380 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773347    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e46743-e695-4a88-9a66-7d724622d04f-cabundle-cert\") pod \"f8e46743-e695-4a88-9a66-7d724622d04f\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") "
Sep 17 22:39:49.773380 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773372    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-model-cache\") pod \"f8e46743-e695-4a88-9a66-7d724622d04f\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") "
Sep 17 22:39:49.773672 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773401    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxwks\" (UniqueName: \"kubernetes.io/projected/f8e46743-e695-4a88-9a66-7d724622d04f-kube-api-access-nxwks\") pod \"f8e46743-e695-4a88-9a66-7d724622d04f\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") "
Sep 17 22:39:49.773672 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773437    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-kserve-provision-location\") pod \"f8e46743-e695-4a88-9a66-7d724622d04f\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") "
Sep 17 22:39:49.773672 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773468    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-home\") pod \"f8e46743-e695-4a88-9a66-7d724622d04f\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") "
Sep 17 22:39:49.773672 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773511    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e46743-e695-4a88-9a66-7d724622d04f-tls-certs\") pod \"f8e46743-e695-4a88-9a66-7d724622d04f\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") "
Sep 17 22:39:49.773672 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773556    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-dshm\") pod \"f8e46743-e695-4a88-9a66-7d724622d04f\" (UID: \"f8e46743-e695-4a88-9a66-7d724622d04f\") "
Sep 17 22:39:49.773944 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773679    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8e46743-e695-4a88-9a66-7d724622d04f" (UID: "f8e46743-e695-4a88-9a66-7d724622d04f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:39:49.773944 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773774    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e46743-e695-4a88-9a66-7d724622d04f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f8e46743-e695-4a88-9a66-7d724622d04f" (UID: "f8e46743-e695-4a88-9a66-7d724622d04f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:39:49.773944 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773843    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8e46743-e695-4a88-9a66-7d724622d04f" (UID: "f8e46743-e695-4a88-9a66-7d724622d04f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:39:49.773944 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773925    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:39:49.774184 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773951    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e46743-e695-4a88-9a66-7d724622d04f-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:39:49.774184 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773968    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:39:49.774184 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.773976    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-home" (OuterVolumeSpecName: "home") pod "f8e46743-e695-4a88-9a66-7d724622d04f" (UID: "f8e46743-e695-4a88-9a66-7d724622d04f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:39:49.775821 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.775798    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-dshm" (OuterVolumeSpecName: "dshm") pod "f8e46743-e695-4a88-9a66-7d724622d04f" (UID: "f8e46743-e695-4a88-9a66-7d724622d04f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:39:49.775936 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.775913    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e46743-e695-4a88-9a66-7d724622d04f-kube-api-access-nxwks" (OuterVolumeSpecName: "kube-api-access-nxwks") pod "f8e46743-e695-4a88-9a66-7d724622d04f" (UID: "f8e46743-e695-4a88-9a66-7d724622d04f"). InnerVolumeSpecName "kube-api-access-nxwks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:39:49.775983 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.775941    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e46743-e695-4a88-9a66-7d724622d04f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8e46743-e695-4a88-9a66-7d724622d04f" (UID: "f8e46743-e695-4a88-9a66-7d724622d04f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:39:49.796432 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.796402    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8e46743-e695-4a88-9a66-7d724622d04f" (UID: "f8e46743-e695-4a88-9a66-7d724622d04f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:39:49.812401 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.812379    2567 generic.go:356] "Generic (PLEG): container finished" podID="f8e46743-e695-4a88-9a66-7d724622d04f" containerID="85359b0d39a8d438519949471604788827a9751add9fd7573d48ab78dfd5bf0e" exitCode=0
Sep 17 22:39:49.812496 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.812426    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"
Sep 17 22:39:49.812496 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.812431    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" event={"ID":"f8e46743-e695-4a88-9a66-7d724622d04f","Type":"ContainerDied","Data":"85359b0d39a8d438519949471604788827a9751add9fd7573d48ab78dfd5bf0e"}
Sep 17 22:39:49.812496 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.812469    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz" event={"ID":"f8e46743-e695-4a88-9a66-7d724622d04f","Type":"ContainerDied","Data":"0515e5fe54f7339253f209b00333bd1a96783b4f6d595c29441d0c314608300f"}
Sep 17 22:39:49.812496 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.812490    2567 scope.go:117] "RemoveContainer" containerID="85359b0d39a8d438519949471604788827a9751add9fd7573d48ab78dfd5bf0e"
Sep 17 22:39:49.820555 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.820541    2567 scope.go:117] "RemoveContainer" containerID="e8e6317aafa537d225ed90e85b95a5c8ef6c8f81b39d93be63af5b45a9070b3d"
Sep 17 22:39:49.827758 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.827744    2567 scope.go:117] "RemoveContainer" containerID="85359b0d39a8d438519949471604788827a9751add9fd7573d48ab78dfd5bf0e"
Sep 17 22:39:49.828031 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:39:49.828014    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85359b0d39a8d438519949471604788827a9751add9fd7573d48ab78dfd5bf0e\": container with ID starting with 85359b0d39a8d438519949471604788827a9751add9fd7573d48ab78dfd5bf0e not found: ID does not exist" containerID="85359b0d39a8d438519949471604788827a9751add9fd7573d48ab78dfd5bf0e"
Sep 17 22:39:49.828080 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.828038    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85359b0d39a8d438519949471604788827a9751add9fd7573d48ab78dfd5bf0e"} err="failed to get container status \"85359b0d39a8d438519949471604788827a9751add9fd7573d48ab78dfd5bf0e\": rpc error: code = NotFound desc = could not find container \"85359b0d39a8d438519949471604788827a9751add9fd7573d48ab78dfd5bf0e\": container with ID starting with 85359b0d39a8d438519949471604788827a9751add9fd7573d48ab78dfd5bf0e not found: ID does not exist"
Sep 17 22:39:49.828080 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.828053    2567 scope.go:117] "RemoveContainer" containerID="e8e6317aafa537d225ed90e85b95a5c8ef6c8f81b39d93be63af5b45a9070b3d"
Sep 17 22:39:49.828289 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:39:49.828273    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e6317aafa537d225ed90e85b95a5c8ef6c8f81b39d93be63af5b45a9070b3d\": container with ID starting with e8e6317aafa537d225ed90e85b95a5c8ef6c8f81b39d93be63af5b45a9070b3d not found: ID does not exist" containerID="e8e6317aafa537d225ed90e85b95a5c8ef6c8f81b39d93be63af5b45a9070b3d"
Sep 17 22:39:49.828337 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.828294    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e6317aafa537d225ed90e85b95a5c8ef6c8f81b39d93be63af5b45a9070b3d"} err="failed to get container status \"e8e6317aafa537d225ed90e85b95a5c8ef6c8f81b39d93be63af5b45a9070b3d\": rpc error: code = NotFound desc = could not find container \"e8e6317aafa537d225ed90e85b95a5c8ef6c8f81b39d93be63af5b45a9070b3d\": container with ID starting with e8e6317aafa537d225ed90e85b95a5c8ef6c8f81b39d93be63af5b45a9070b3d not found: ID does not exist"
Sep 17 22:39:49.833043 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.833020    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"]
Sep 17 22:39:49.836786 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.836767    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5c7db44f7b-4xpmz"]
Sep 17 22:39:49.875060 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.875039    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:39:49.875060 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.875056    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:39:49.875174 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.875066    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8e46743-e695-4a88-9a66-7d724622d04f-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:39:49.875174 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.875074    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8e46743-e695-4a88-9a66-7d724622d04f-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:39:49.875174 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:49.875082    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxwks\" (UniqueName: \"kubernetes.io/projected/f8e46743-e695-4a88-9a66-7d724622d04f-kube-api-access-nxwks\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:39:51.639743 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:51.639710    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" path="/var/lib/kubelet/pods/f8e46743-e695-4a88-9a66-7d724622d04f/volumes"
Sep 17 22:39:51.708782 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:39:51.708761    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"
Sep 17 22:40:38.288857 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.288767    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"]
Sep 17 22:40:38.289458 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.289418    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" containerName="storage-initializer"
Sep 17 22:40:38.289458 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.289437    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" containerName="storage-initializer"
Sep 17 22:40:38.289567 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.289493    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" containerName="main"
Sep 17 22:40:38.289567 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.289502    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" containerName="main"
Sep 17 22:40:38.289665 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.289620    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e46743-e695-4a88-9a66-7d724622d04f" containerName="main"
Sep 17 22:40:38.294695 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.294671    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"
Sep 17 22:40:38.298284 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.298264    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 22:40:38.298284 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.298272    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-qv8fc\""
Sep 17 22:40:38.298474 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.298296    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-c74vp\""
Sep 17 22:40:38.298474 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.298326    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 22:40:38.298474 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.298327    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 22:40:38.302042 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.301764    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"]
Sep 17 22:40:38.402216 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.402189    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8f3e20-ec9c-443a-b601-81ad6c9b5841-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d\" (UID: \"aa8f3e20-ec9c-443a-b601-81ad6c9b5841\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"
Sep 17 22:40:38.402346 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.402222    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffgb2\" (UniqueName: \"kubernetes.io/projected/aa8f3e20-ec9c-443a-b601-81ad6c9b5841-kube-api-access-ffgb2\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d\" (UID: \"aa8f3e20-ec9c-443a-b601-81ad6c9b5841\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"
Sep 17 22:40:38.502981 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.502959    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8f3e20-ec9c-443a-b601-81ad6c9b5841-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d\" (UID: \"aa8f3e20-ec9c-443a-b601-81ad6c9b5841\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"
Sep 17 22:40:38.503104 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.502986    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffgb2\" (UniqueName: \"kubernetes.io/projected/aa8f3e20-ec9c-443a-b601-81ad6c9b5841-kube-api-access-ffgb2\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d\" (UID: \"aa8f3e20-ec9c-443a-b601-81ad6c9b5841\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"
Sep 17 22:40:38.505395 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.505371    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8f3e20-ec9c-443a-b601-81ad6c9b5841-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d\" (UID: \"aa8f3e20-ec9c-443a-b601-81ad6c9b5841\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"
Sep 17 22:40:38.512817 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.512796    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffgb2\" (UniqueName: \"kubernetes.io/projected/aa8f3e20-ec9c-443a-b601-81ad6c9b5841-kube-api-access-ffgb2\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d\" (UID: \"aa8f3e20-ec9c-443a-b601-81ad6c9b5841\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"
Sep 17 22:40:38.604878 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.604822    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"
Sep 17 22:40:38.874577 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.874431    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"]
Sep 17 22:40:38.877259 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:40:38.877230    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8f3e20_ec9c_443a_b601_81ad6c9b5841.slice/crio-a2d4cb57d6b4d761594e9a32c2666c272a7c5a8fb5615142385514e018aa0083 WatchSource:0}: Error finding container a2d4cb57d6b4d761594e9a32c2666c272a7c5a8fb5615142385514e018aa0083: Status 404 returned error can't find the container with id a2d4cb57d6b4d761594e9a32c2666c272a7c5a8fb5615142385514e018aa0083
Sep 17 22:40:38.983022 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.982961    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d" event={"ID":"aa8f3e20-ec9c-443a-b601-81ad6c9b5841","Type":"ContainerStarted","Data":"f001433876f3bc8db180cea780668f3cc68bfcc457e14a10abc0290be5b4e897"}
Sep 17 22:40:38.983193 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.983053    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d" event={"ID":"aa8f3e20-ec9c-443a-b601-81ad6c9b5841","Type":"ContainerStarted","Data":"a2d4cb57d6b4d761594e9a32c2666c272a7c5a8fb5615142385514e018aa0083"}
Sep 17 22:40:38.983754 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:40:38.983727    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"
Sep 17 22:41:10.992363 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:41:10.992335    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"
Sep 17 22:41:11.012566 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:41:11.012518    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d" podStartSLOduration=33.01250257 podStartE2EDuration="33.01250257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:40:38.998767484 +0000 UTC m=+2516.007697025" watchObservedRunningTime="2026-09-17 22:41:11.01250257 +0000 UTC m=+2548.021432110"
Sep 17 22:41:25.094903 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:41:25.094856    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"]
Sep 17 22:41:31.708598 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:41:31.708558    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2" podUID="3c28debd-4c15-4fa3-b4e4-e7b9298df064" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:41:40.113949 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:41:40.113884    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2" podUID="3c28debd-4c15-4fa3-b4e4-e7b9298df064" containerName="main" containerID="cri-o://91ae6b7e8fae2c36b45929e4ae11bb6c1e1559078fae79c293cd848c3c448550" gracePeriod=45
Sep 17 22:41:41.709189 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:41:41.709150    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2" podUID="3c28debd-4c15-4fa3-b4e4-e7b9298df064" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:41:51.709028 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:41:51.708973    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2" podUID="3c28debd-4c15-4fa3-b4e4-e7b9298df064" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:41:51.709582 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:41:51.709105    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"
Sep 17 22:42:01.709151 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:01.709116    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2" podUID="3c28debd-4c15-4fa3-b4e4-e7b9298df064" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:42:10.305836 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:10.305644    2567 generic.go:356] "Generic (PLEG): container finished" podID="3c28debd-4c15-4fa3-b4e4-e7b9298df064" containerID="91ae6b7e8fae2c36b45929e4ae11bb6c1e1559078fae79c293cd848c3c448550" exitCode=0
Sep 17 22:42:10.307047 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:10.305921    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2" event={"ID":"3c28debd-4c15-4fa3-b4e4-e7b9298df064","Type":"ContainerDied","Data":"91ae6b7e8fae2c36b45929e4ae11bb6c1e1559078fae79c293cd848c3c448550"}
Sep 17 22:42:10.357469 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:10.357448    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"
Sep 17 22:42:10.449611 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:10.449551    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c28debd-4c15-4fa3-b4e4-e7b9298df064-tls-certs\") pod \"3c28debd-4c15-4fa3-b4e4-e7b9298df064\" (UID: \"3c28debd-4c15-4fa3-b4e4-e7b9298df064\") "
Sep 17 22:42:10.449611 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:10.449598    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r5lj\" (UniqueName: \"kubernetes.io/projected/3c28debd-4c15-4fa3-b4e4-e7b9298df064-kube-api-access-9r5lj\") pod \"3c28debd-4c15-4fa3-b4e4-e7b9298df064\" (UID: \"3c28debd-4c15-4fa3-b4e4-e7b9298df064\") "
Sep 17 22:42:10.451680 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:10.451658    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c28debd-4c15-4fa3-b4e4-e7b9298df064-kube-api-access-9r5lj" (OuterVolumeSpecName: "kube-api-access-9r5lj") pod "3c28debd-4c15-4fa3-b4e4-e7b9298df064" (UID: "3c28debd-4c15-4fa3-b4e4-e7b9298df064"). InnerVolumeSpecName "kube-api-access-9r5lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:42:10.451765 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:10.451714    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c28debd-4c15-4fa3-b4e4-e7b9298df064-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c28debd-4c15-4fa3-b4e4-e7b9298df064" (UID: "3c28debd-4c15-4fa3-b4e4-e7b9298df064"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:42:10.550274 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:10.550244    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r5lj\" (UniqueName: \"kubernetes.io/projected/3c28debd-4c15-4fa3-b4e4-e7b9298df064-kube-api-access-9r5lj\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:42:10.550274 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:10.550270    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c28debd-4c15-4fa3-b4e4-e7b9298df064-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:42:11.310159 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:11.310130    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"
Sep 17 22:42:11.310159 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:11.310140    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2" event={"ID":"3c28debd-4c15-4fa3-b4e4-e7b9298df064","Type":"ContainerDied","Data":"d932d9edc72843d0943274bf9d8b89a17b3cbc7d42d8845bff2faebd88ec06f8"}
Sep 17 22:42:11.310618 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:11.310195    2567 scope.go:117] "RemoveContainer" containerID="91ae6b7e8fae2c36b45929e4ae11bb6c1e1559078fae79c293cd848c3c448550"
Sep 17 22:42:11.331812 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:11.331787    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"]
Sep 17 22:42:11.335144 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:11.335124    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58bkhq2"]
Sep 17 22:42:11.638895 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:11.638824    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c28debd-4c15-4fa3-b4e4-e7b9298df064" path="/var/lib/kubelet/pods/3c28debd-4c15-4fa3-b4e4-e7b9298df064/volumes"
Sep 17 22:42:15.259680 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.259653    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"]
Sep 17 22:42:15.260075 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.259934    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c28debd-4c15-4fa3-b4e4-e7b9298df064" containerName="main"
Sep 17 22:42:15.260075 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.259943    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c28debd-4c15-4fa3-b4e4-e7b9298df064" containerName="main"
Sep 17 22:42:15.260075 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.259989    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c28debd-4c15-4fa3-b4e4-e7b9298df064" containerName="main"
Sep 17 22:42:15.264302 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.264286    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.266926 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.266904    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 17 22:42:15.266926 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.266921    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 22:42:15.267119 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.266977    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-xv96r\""
Sep 17 22:42:15.267278 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.267263    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 22:42:15.275849 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.275829    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"]
Sep 17 22:42:15.379816 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.379790    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-data\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.379932 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.379822    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-credential-socket\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.379932 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.379842    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-workload-socket\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.379932 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.379860    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gllj\" (UniqueName: \"kubernetes.io/projected/3a570df0-dc46-42da-ae11-c48773df5ff9-kube-api-access-8gllj\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.379932 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.379923    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-envoy\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.380133 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.379948    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-token\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.380133 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.379984    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-workload-certs\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.380133 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.380024    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a570df0-dc46-42da-ae11-c48773df5ff9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.380133 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.380102    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-podinfo\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.480709 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.480677    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-podinfo\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.480805 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.480721    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-data\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.480805 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.480748    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-credential-socket\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.480805 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.480772    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-workload-socket\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.480805 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.480795    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gllj\" (UniqueName: \"kubernetes.io/projected/3a570df0-dc46-42da-ae11-c48773df5ff9-kube-api-access-8gllj\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.481040 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.480824    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-envoy\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.481040 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.480856    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-token\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.481040 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.480904    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-workload-certs\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.481040 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.480928    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a570df0-dc46-42da-ae11-c48773df5ff9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.481286 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.481183    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-data\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.481340 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.481298    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-workload-socket\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.481395 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.481354    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-credential-socket\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.481539 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.481519    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-workload-certs\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.481645 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.481627    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a570df0-dc46-42da-ae11-c48773df5ff9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.483115 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.483094    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-envoy\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.483246 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.483229    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-podinfo\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.490096 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.490079    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-token\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.493881 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.493865    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gllj\" (UniqueName: \"kubernetes.io/projected/3a570df0-dc46-42da-ae11-c48773df5ff9-kube-api-access-8gllj\") pod \"router-gateway-1-openshift-default-5d8488d74d-z9bzq\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.573923 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.573870    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:15.852302 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.852140    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"]
Sep 17 22:42:15.854926 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:42:15.854899    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a570df0_dc46_42da_ae11_c48773df5ff9.slice/crio-87a7e639979e86a8f1737bc7ce3e5dbab74a4fdd7e1623946192477cd9c72f78 WatchSource:0}: Error finding container 87a7e639979e86a8f1737bc7ce3e5dbab74a4fdd7e1623946192477cd9c72f78: Status 404 returned error can't find the container with id 87a7e639979e86a8f1737bc7ce3e5dbab74a4fdd7e1623946192477cd9c72f78
Sep 17 22:42:15.856867 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.856832    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:42:15.856972 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.856902    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:42:15.856972 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:15.856942    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:42:16.329877 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:16.329842    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq" event={"ID":"3a570df0-dc46-42da-ae11-c48773df5ff9","Type":"ContainerStarted","Data":"57b65790418f96689930601024bd83cc21efd56ca640a004b6b0cb201906eca5"}
Sep 17 22:42:16.330261 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:16.329894    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq" event={"ID":"3a570df0-dc46-42da-ae11-c48773df5ff9","Type":"ContainerStarted","Data":"87a7e639979e86a8f1737bc7ce3e5dbab74a4fdd7e1623946192477cd9c72f78"}
Sep 17 22:42:16.349102 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:16.349059    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq" podStartSLOduration=1.349045044 podStartE2EDuration="1.349045044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:42:16.348116039 +0000 UTC m=+2613.357045580" watchObservedRunningTime="2026-09-17 22:42:16.349045044 +0000 UTC m=+2613.357974584"
Sep 17 22:42:16.574057 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:16.574027    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:16.578189 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:16.578168    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:17.333154 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:17.333121    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:17.334171 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:17.334150    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:27.377105 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:27.377070    2567 generic.go:356] "Generic (PLEG): container finished" podID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerID="57b65790418f96689930601024bd83cc21efd56ca640a004b6b0cb201906eca5" exitCode=0
Sep 17 22:42:27.377105 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:27.377105    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq" event={"ID":"3a570df0-dc46-42da-ae11-c48773df5ff9","Type":"ContainerDied","Data":"57b65790418f96689930601024bd83cc21efd56ca640a004b6b0cb201906eca5"}
Sep 17 22:42:27.377532 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:27.377392    2567 scope.go:117] "RemoveContainer" containerID="57b65790418f96689930601024bd83cc21efd56ca640a004b6b0cb201906eca5"
Sep 17 22:42:27.378342 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:27.378314    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:42:27.378398 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:27.378383    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:42:27.378439 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:27.378414    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:42:28.382756 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:28.382721    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq" event={"ID":"3a570df0-dc46-42da-ae11-c48773df5ff9","Type":"ContainerStarted","Data":"943087284abaad2fc1c199c50ce86ff9ff22c27626fb4f6f256bb6a02031d6cb"}
Sep 17 22:42:28.574084 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:28.574043    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:28.575163 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:28.575133    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq" podUID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 22:42:29.575282 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:29.575242    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq" podUID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 22:42:30.574301 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:30.574256    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq" podUID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 22:42:31.574511 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:31.574465    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq" podUID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 22:42:32.332991 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:32.332953    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:32.575220 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:32.575180    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq" podUID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 17 22:42:33.578059 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:33.578027    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:33.579150 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:33.579131    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:42:49.909780 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:49.909747    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"]
Sep 17 22:42:50.993600 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:42:50.993549    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d" podUID="aa8f3e20-ec9c-443a-b601-81ad6c9b5841" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:43:00.992105 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:00.992067    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d" podUID="aa8f3e20-ec9c-443a-b601-81ad6c9b5841" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:43:02.849740 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:02.849709    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"]
Sep 17 22:43:02.852269 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:02.852250    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:02.854544 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:02.854517    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:43:02.854673 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:02.854557    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 22:43:02.863836 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:02.863810    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"]
Sep 17 22:43:02.933358 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:02.933322    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17d72d8-89fa-4aee-a513-c75b5aabe877-cabundle-cert\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:02.933358 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:02.933366    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-home\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:02.933572 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:02.933385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-dshm\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:02.933572 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:02.933400    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pmkt\" (UniqueName: \"kubernetes.io/projected/b17d72d8-89fa-4aee-a513-c75b5aabe877-kube-api-access-8pmkt\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:02.933572 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:02.933442    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-tmp-dir\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:02.933572 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:02.933462    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-model-cache\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:02.933572 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:02.933501    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-kserve-provision-location\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:02.933572 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:02.933522    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17d72d8-89fa-4aee-a513-c75b5aabe877-tls-certs\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.034362 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.034320    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-home\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.034362 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.034364    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-dshm\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.034596 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.034380    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmkt\" (UniqueName: \"kubernetes.io/projected/b17d72d8-89fa-4aee-a513-c75b5aabe877-kube-api-access-8pmkt\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.034596 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.034511    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-tmp-dir\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.034596 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.034554    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-model-cache\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.034721 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.034602    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-kserve-provision-location\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.034721 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.034629    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17d72d8-89fa-4aee-a513-c75b5aabe877-tls-certs\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.034808 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.034718    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17d72d8-89fa-4aee-a513-c75b5aabe877-cabundle-cert\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.034808 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.034794    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-home\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.034912 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.034883    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-tmp-dir\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.035014 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.034968    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-model-cache\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.035145 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.035122    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-kserve-provision-location\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.035542 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.035521    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17d72d8-89fa-4aee-a513-c75b5aabe877-cabundle-cert\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.036842 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.036818    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-dshm\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.037118 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.037099    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17d72d8-89fa-4aee-a513-c75b5aabe877-tls-certs\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.042245 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.042226    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmkt\" (UniqueName: \"kubernetes.io/projected/b17d72d8-89fa-4aee-a513-c75b5aabe877-kube-api-access-8pmkt\") pod \"stop-feature-test-kserve-85486f684b-wgvtl\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.161813 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.161718    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:03.437738 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.437540    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"]
Sep 17 22:43:03.440328 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:43:03.440288    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17d72d8_89fa_4aee_a513_c75b5aabe877.slice/crio-c8d0a3bca7f242fa81030eae3c44458f7075e1357222c1dbd0a7b13e931d7045 WatchSource:0}: Error finding container c8d0a3bca7f242fa81030eae3c44458f7075e1357222c1dbd0a7b13e931d7045: Status 404 returned error can't find the container with id c8d0a3bca7f242fa81030eae3c44458f7075e1357222c1dbd0a7b13e931d7045
Sep 17 22:43:03.503591 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.503557    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" event={"ID":"b17d72d8-89fa-4aee-a513-c75b5aabe877","Type":"ContainerStarted","Data":"08c5de7763a82c412566a2a0e239656a0ecb4a524d21d79c8399f6b4832d5366"}
Sep 17 22:43:03.503710 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:03.503622    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" event={"ID":"b17d72d8-89fa-4aee-a513-c75b5aabe877","Type":"ContainerStarted","Data":"c8d0a3bca7f242fa81030eae3c44458f7075e1357222c1dbd0a7b13e931d7045"}
Sep 17 22:43:04.928505 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:04.928454    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d" podUID="aa8f3e20-ec9c-443a-b601-81ad6c9b5841" containerName="main" containerID="cri-o://f001433876f3bc8db180cea780668f3cc68bfcc457e14a10abc0290be5b4e897" gracePeriod=45
Sep 17 22:43:06.516482 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:06.516448    2567 generic.go:356] "Generic (PLEG): container finished" podID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerID="08c5de7763a82c412566a2a0e239656a0ecb4a524d21d79c8399f6b4832d5366" exitCode=0
Sep 17 22:43:06.516839 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:06.516520    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" event={"ID":"b17d72d8-89fa-4aee-a513-c75b5aabe877","Type":"ContainerDied","Data":"08c5de7763a82c412566a2a0e239656a0ecb4a524d21d79c8399f6b4832d5366"}
Sep 17 22:43:07.522440 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:07.522405    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" event={"ID":"b17d72d8-89fa-4aee-a513-c75b5aabe877","Type":"ContainerStarted","Data":"9c5820c65c924bfd90269496f4aee60d0ab02dc748d232573234bae5b8c669c6"}
Sep 17 22:43:07.543597 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:07.543551    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" podStartSLOduration=3.504905639 podStartE2EDuration="5.543535952s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.038630313s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:43:07.54155619 +0000 UTC m=+2664.550485732" watchObservedRunningTime="2026-09-17 22:43:07.543535952 +0000 UTC m=+2664.552465493"
Sep 17 22:43:08.162303 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:08.162261    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:10.992788 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:10.992752    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d" podUID="aa8f3e20-ec9c-443a-b601-81ad6c9b5841" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:43:10.993246 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:10.992865    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"
Sep 17 22:43:13.162529 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:13.162497    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:43:13.164109 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:13.164081    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 17 22:43:20.992504 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:20.992465    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d" podUID="aa8f3e20-ec9c-443a-b601-81ad6c9b5841" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:43:23.162670 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:23.162624    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 17 22:43:30.992313 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:30.992274    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d" podUID="aa8f3e20-ec9c-443a-b601-81ad6c9b5841" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:43:33.163018 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:33.162965    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 17 22:43:35.167892 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.167857    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"
Sep 17 22:43:35.196039 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.195949    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8f3e20-ec9c-443a-b601-81ad6c9b5841-tls-certs\") pod \"aa8f3e20-ec9c-443a-b601-81ad6c9b5841\" (UID: \"aa8f3e20-ec9c-443a-b601-81ad6c9b5841\") "
Sep 17 22:43:35.196039 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.196012    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffgb2\" (UniqueName: \"kubernetes.io/projected/aa8f3e20-ec9c-443a-b601-81ad6c9b5841-kube-api-access-ffgb2\") pod \"aa8f3e20-ec9c-443a-b601-81ad6c9b5841\" (UID: \"aa8f3e20-ec9c-443a-b601-81ad6c9b5841\") "
Sep 17 22:43:35.198160 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.198132    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa8f3e20-ec9c-443a-b601-81ad6c9b5841-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa8f3e20-ec9c-443a-b601-81ad6c9b5841" (UID: "aa8f3e20-ec9c-443a-b601-81ad6c9b5841"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:43:35.198306 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.198195    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8f3e20-ec9c-443a-b601-81ad6c9b5841-kube-api-access-ffgb2" (OuterVolumeSpecName: "kube-api-access-ffgb2") pod "aa8f3e20-ec9c-443a-b601-81ad6c9b5841" (UID: "aa8f3e20-ec9c-443a-b601-81ad6c9b5841"). InnerVolumeSpecName "kube-api-access-ffgb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:43:35.297521 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.297443    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa8f3e20-ec9c-443a-b601-81ad6c9b5841-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:43:35.297521 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.297487    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffgb2\" (UniqueName: \"kubernetes.io/projected/aa8f3e20-ec9c-443a-b601-81ad6c9b5841-kube-api-access-ffgb2\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:43:35.639079 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.639042    2567 generic.go:356] "Generic (PLEG): container finished" podID="aa8f3e20-ec9c-443a-b601-81ad6c9b5841" containerID="f001433876f3bc8db180cea780668f3cc68bfcc457e14a10abc0290be5b4e897" exitCode=0
Sep 17 22:43:35.639246 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.639133    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"
Sep 17 22:43:35.639436 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.639414    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d" event={"ID":"aa8f3e20-ec9c-443a-b601-81ad6c9b5841","Type":"ContainerDied","Data":"f001433876f3bc8db180cea780668f3cc68bfcc457e14a10abc0290be5b4e897"}
Sep 17 22:43:35.639478 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.639460    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d" event={"ID":"aa8f3e20-ec9c-443a-b601-81ad6c9b5841","Type":"ContainerDied","Data":"a2d4cb57d6b4d761594e9a32c2666c272a7c5a8fb5615142385514e018aa0083"}
Sep 17 22:43:35.639522 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.639491    2567 scope.go:117] "RemoveContainer" containerID="f001433876f3bc8db180cea780668f3cc68bfcc457e14a10abc0290be5b4e897"
Sep 17 22:43:35.648889 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.648871    2567 scope.go:117] "RemoveContainer" containerID="f001433876f3bc8db180cea780668f3cc68bfcc457e14a10abc0290be5b4e897"
Sep 17 22:43:35.649202 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:43:35.649184    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f001433876f3bc8db180cea780668f3cc68bfcc457e14a10abc0290be5b4e897\": container with ID starting with f001433876f3bc8db180cea780668f3cc68bfcc457e14a10abc0290be5b4e897 not found: ID does not exist" containerID="f001433876f3bc8db180cea780668f3cc68bfcc457e14a10abc0290be5b4e897"
Sep 17 22:43:35.649285 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.649209    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f001433876f3bc8db180cea780668f3cc68bfcc457e14a10abc0290be5b4e897"} err="failed to get container status \"f001433876f3bc8db180cea780668f3cc68bfcc457e14a10abc0290be5b4e897\": rpc error: code = NotFound desc = could not find container \"f001433876f3bc8db180cea780668f3cc68bfcc457e14a10abc0290be5b4e897\": container with ID starting with f001433876f3bc8db180cea780668f3cc68bfcc457e14a10abc0290be5b4e897 not found: ID does not exist"
Sep 17 22:43:35.662473 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.662451    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"]
Sep 17 22:43:35.666727 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:35.666707    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-frf2d"]
Sep 17 22:43:37.639834 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:37.639801    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa8f3e20-ec9c-443a-b601-81ad6c9b5841" path="/var/lib/kubelet/pods/aa8f3e20-ec9c-443a-b601-81ad6c9b5841/volumes"
Sep 17 22:43:43.162583 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:43.162538    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 17 22:43:53.162713 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:43:53.162671    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 17 22:44:03.162395 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:44:03.162353    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 17 22:44:13.162770 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:44:13.162724    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 17 22:44:23.162649 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:44:23.162607    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 17 22:44:33.162560 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:44:33.162519    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 17 22:44:43.163046 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:44:43.162968    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 17 22:44:53.172146 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:44:53.172116    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:44:53.179726 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:44:53.179698    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:44:55.002111 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:44:55.002070    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"]
Sep 17 22:45:10.024664 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.024552    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main" containerID="cri-o://9c5820c65c924bfd90269496f4aee60d0ab02dc748d232573234bae5b8c669c6" gracePeriod=45
Sep 17 22:45:10.172984 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.172945    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 17 22:45:10.763246 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.763222    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:45:10.816867 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.816808    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-dshm\") pod \"b17d72d8-89fa-4aee-a513-c75b5aabe877\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") "
Sep 17 22:45:10.816867 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.816850    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pmkt\" (UniqueName: \"kubernetes.io/projected/b17d72d8-89fa-4aee-a513-c75b5aabe877-kube-api-access-8pmkt\") pod \"b17d72d8-89fa-4aee-a513-c75b5aabe877\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") "
Sep 17 22:45:10.817037 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.816880    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-home\") pod \"b17d72d8-89fa-4aee-a513-c75b5aabe877\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") "
Sep 17 22:45:10.817037 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.816902    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17d72d8-89fa-4aee-a513-c75b5aabe877-tls-certs\") pod \"b17d72d8-89fa-4aee-a513-c75b5aabe877\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") "
Sep 17 22:45:10.817037 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.816961    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-tmp-dir\") pod \"b17d72d8-89fa-4aee-a513-c75b5aabe877\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") "
Sep 17 22:45:10.817037 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.817023    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17d72d8-89fa-4aee-a513-c75b5aabe877-cabundle-cert\") pod \"b17d72d8-89fa-4aee-a513-c75b5aabe877\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") "
Sep 17 22:45:10.817242 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.817048    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-model-cache\") pod \"b17d72d8-89fa-4aee-a513-c75b5aabe877\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") "
Sep 17 22:45:10.817242 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.817084    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-kserve-provision-location\") pod \"b17d72d8-89fa-4aee-a513-c75b5aabe877\" (UID: \"b17d72d8-89fa-4aee-a513-c75b5aabe877\") "
Sep 17 22:45:10.817587 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.817480    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17d72d8-89fa-4aee-a513-c75b5aabe877-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b17d72d8-89fa-4aee-a513-c75b5aabe877" (UID: "b17d72d8-89fa-4aee-a513-c75b5aabe877"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:45:10.817770 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.817741    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-home" (OuterVolumeSpecName: "home") pod "b17d72d8-89fa-4aee-a513-c75b5aabe877" (UID: "b17d72d8-89fa-4aee-a513-c75b5aabe877"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:45:10.817940 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.817914    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-model-cache" (OuterVolumeSpecName: "model-cache") pod "b17d72d8-89fa-4aee-a513-c75b5aabe877" (UID: "b17d72d8-89fa-4aee-a513-c75b5aabe877"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:45:10.819085 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.819059    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17d72d8-89fa-4aee-a513-c75b5aabe877-kube-api-access-8pmkt" (OuterVolumeSpecName: "kube-api-access-8pmkt") pod "b17d72d8-89fa-4aee-a513-c75b5aabe877" (UID: "b17d72d8-89fa-4aee-a513-c75b5aabe877"). InnerVolumeSpecName "kube-api-access-8pmkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:45:10.819188 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.819085    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b17d72d8-89fa-4aee-a513-c75b5aabe877-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b17d72d8-89fa-4aee-a513-c75b5aabe877" (UID: "b17d72d8-89fa-4aee-a513-c75b5aabe877"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:45:10.819253 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.819203    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-dshm" (OuterVolumeSpecName: "dshm") pod "b17d72d8-89fa-4aee-a513-c75b5aabe877" (UID: "b17d72d8-89fa-4aee-a513-c75b5aabe877"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:45:10.829513 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.829487    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b17d72d8-89fa-4aee-a513-c75b5aabe877" (UID: "b17d72d8-89fa-4aee-a513-c75b5aabe877"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:45:10.839454 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.839433    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b17d72d8-89fa-4aee-a513-c75b5aabe877" (UID: "b17d72d8-89fa-4aee-a513-c75b5aabe877"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:45:10.918321 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.918301    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:10.918321 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.918321    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b17d72d8-89fa-4aee-a513-c75b5aabe877-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:10.918420 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.918329    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:10.918420 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.918338    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:10.918420 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.918346    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:10.918420 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.918354    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pmkt\" (UniqueName: \"kubernetes.io/projected/b17d72d8-89fa-4aee-a513-c75b5aabe877-kube-api-access-8pmkt\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:10.918420 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.918361    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17d72d8-89fa-4aee-a513-c75b5aabe877-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:10.918420 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:10.918368    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17d72d8-89fa-4aee-a513-c75b5aabe877-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:11.004659 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:11.004632    2567 generic.go:356] "Generic (PLEG): container finished" podID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerID="9c5820c65c924bfd90269496f4aee60d0ab02dc748d232573234bae5b8c669c6" exitCode=0
Sep 17 22:45:11.004778 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:11.004683    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"
Sep 17 22:45:11.004778 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:11.004714    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" event={"ID":"b17d72d8-89fa-4aee-a513-c75b5aabe877","Type":"ContainerDied","Data":"9c5820c65c924bfd90269496f4aee60d0ab02dc748d232573234bae5b8c669c6"}
Sep 17 22:45:11.004778 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:11.004776    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl" event={"ID":"b17d72d8-89fa-4aee-a513-c75b5aabe877","Type":"ContainerDied","Data":"c8d0a3bca7f242fa81030eae3c44458f7075e1357222c1dbd0a7b13e931d7045"}
Sep 17 22:45:11.004957 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:11.004805    2567 scope.go:117] "RemoveContainer" containerID="9c5820c65c924bfd90269496f4aee60d0ab02dc748d232573234bae5b8c669c6"
Sep 17 22:45:11.013406 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:11.013391    2567 scope.go:117] "RemoveContainer" containerID="08c5de7763a82c412566a2a0e239656a0ecb4a524d21d79c8399f6b4832d5366"
Sep 17 22:45:11.020218 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:11.020198    2567 scope.go:117] "RemoveContainer" containerID="9c5820c65c924bfd90269496f4aee60d0ab02dc748d232573234bae5b8c669c6"
Sep 17 22:45:11.020481 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:45:11.020461    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5820c65c924bfd90269496f4aee60d0ab02dc748d232573234bae5b8c669c6\": container with ID starting with 9c5820c65c924bfd90269496f4aee60d0ab02dc748d232573234bae5b8c669c6 not found: ID does not exist" containerID="9c5820c65c924bfd90269496f4aee60d0ab02dc748d232573234bae5b8c669c6"
Sep 17 22:45:11.020550 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:11.020489    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5820c65c924bfd90269496f4aee60d0ab02dc748d232573234bae5b8c669c6"} err="failed to get container status \"9c5820c65c924bfd90269496f4aee60d0ab02dc748d232573234bae5b8c669c6\": rpc error: code = NotFound desc = could not find container \"9c5820c65c924bfd90269496f4aee60d0ab02dc748d232573234bae5b8c669c6\": container with ID starting with 9c5820c65c924bfd90269496f4aee60d0ab02dc748d232573234bae5b8c669c6 not found: ID does not exist"
Sep 17 22:45:11.020550 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:11.020504    2567 scope.go:117] "RemoveContainer" containerID="08c5de7763a82c412566a2a0e239656a0ecb4a524d21d79c8399f6b4832d5366"
Sep 17 22:45:11.020727 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:45:11.020711    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c5de7763a82c412566a2a0e239656a0ecb4a524d21d79c8399f6b4832d5366\": container with ID starting with 08c5de7763a82c412566a2a0e239656a0ecb4a524d21d79c8399f6b4832d5366 not found: ID does not exist" containerID="08c5de7763a82c412566a2a0e239656a0ecb4a524d21d79c8399f6b4832d5366"
Sep 17 22:45:11.020769 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:11.020733    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c5de7763a82c412566a2a0e239656a0ecb4a524d21d79c8399f6b4832d5366"} err="failed to get container status \"08c5de7763a82c412566a2a0e239656a0ecb4a524d21d79c8399f6b4832d5366\": rpc error: code = NotFound desc = could not find container \"08c5de7763a82c412566a2a0e239656a0ecb4a524d21d79c8399f6b4832d5366\": container with ID starting with 08c5de7763a82c412566a2a0e239656a0ecb4a524d21d79c8399f6b4832d5366 not found: ID does not exist"
Sep 17 22:45:11.026570 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:11.026550    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"]
Sep 17 22:45:11.029920 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:11.029901    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-85486f684b-wgvtl"]
Sep 17 22:45:11.639564 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:11.639533    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" path="/var/lib/kubelet/pods/b17d72d8-89fa-4aee-a513-c75b5aabe877/volumes"
Sep 17 22:45:29.150539 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:29.150504    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"]
Sep 17 22:45:29.150911 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:29.150731    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq" podUID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerName="istio-proxy" containerID="cri-o://943087284abaad2fc1c199c50ce86ff9ff22c27626fb4f6f256bb6a02031d6cb" gracePeriod=30
Sep 17 22:45:32.423170 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.423129    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"]
Sep 17 22:45:32.423618 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.423484    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa8f3e20-ec9c-443a-b601-81ad6c9b5841" containerName="main"
Sep 17 22:45:32.423618 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.423500    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa8f3e20-ec9c-443a-b601-81ad6c9b5841" containerName="main"
Sep 17 22:45:32.423618 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.423528    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main"
Sep 17 22:45:32.423618 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.423536    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main"
Sep 17 22:45:32.423618 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.423551    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="storage-initializer"
Sep 17 22:45:32.423618 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.423559    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="storage-initializer"
Sep 17 22:45:32.423915 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.423622    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa8f3e20-ec9c-443a-b601-81ad6c9b5841" containerName="main"
Sep 17 22:45:32.423915 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.423636    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b17d72d8-89fa-4aee-a513-c75b5aabe877" containerName="main"
Sep 17 22:45:32.425617 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.425597    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.428091 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.428060    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 22:45:32.428225 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.428208    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-dmgrv\""
Sep 17 22:45:32.428511 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.428492    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:45:32.428613 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.428551    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 22:45:32.428758 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.428744    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 22:45:32.428966 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.428953    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 22:45:32.442404 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.442383    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"]
Sep 17 22:45:32.453033 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.453013    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f438510f-56a1-47f8-9f7d-f6dc72b7d874-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.453127 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.453041    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.453127 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.453061    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.453127 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.453081    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f438510f-56a1-47f8-9f7d-f6dc72b7d874-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.453237 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.453134    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.453237 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.453173    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.453237 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.453199    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blq2f\" (UniqueName: \"kubernetes.io/projected/f438510f-56a1-47f8-9f7d-f6dc72b7d874-kube-api-access-blq2f\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.453332 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.453236    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.553897 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.553878    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f438510f-56a1-47f8-9f7d-f6dc72b7d874-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.554028 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.553907    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.554028 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.553927    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.554028 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.553941    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f438510f-56a1-47f8-9f7d-f6dc72b7d874-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.554028 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.553961    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.554028 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.554014    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.554255 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.554052    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blq2f\" (UniqueName: \"kubernetes.io/projected/f438510f-56a1-47f8-9f7d-f6dc72b7d874-kube-api-access-blq2f\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.554255 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.554085    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.554386 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.554359    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.554386 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.554381    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.554509 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.554458    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.554509 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.554485    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.554721 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.554696    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f438510f-56a1-47f8-9f7d-f6dc72b7d874-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.556334 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.556315    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.556597 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.556578    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f438510f-56a1-47f8-9f7d-f6dc72b7d874-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.561324 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.561297    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blq2f\" (UniqueName: \"kubernetes.io/projected/f438510f-56a1-47f8-9f7d-f6dc72b7d874-kube-api-access-blq2f\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.637591 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.637561    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"]
Sep 17 22:45:32.640316 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.640302    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"
Sep 17 22:45:32.642484 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.642465    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-mzkkm\""
Sep 17 22:45:32.649073 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.649053    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"]
Sep 17 22:45:32.736406 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.736382    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:32.755393 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.755370    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71ece96f-bf47-4f8f-9c40-49e873c87b3e-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr\" (UID: \"71ece96f-bf47-4f8f-9c40-49e873c87b3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"
Sep 17 22:45:32.755484 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.755411    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcw56\" (UniqueName: \"kubernetes.io/projected/71ece96f-bf47-4f8f-9c40-49e873c87b3e-kube-api-access-bcw56\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr\" (UID: \"71ece96f-bf47-4f8f-9c40-49e873c87b3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"
Sep 17 22:45:32.855842 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.855812    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71ece96f-bf47-4f8f-9c40-49e873c87b3e-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr\" (UID: \"71ece96f-bf47-4f8f-9c40-49e873c87b3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"
Sep 17 22:45:32.855976 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.855855    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcw56\" (UniqueName: \"kubernetes.io/projected/71ece96f-bf47-4f8f-9c40-49e873c87b3e-kube-api-access-bcw56\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr\" (UID: \"71ece96f-bf47-4f8f-9c40-49e873c87b3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"
Sep 17 22:45:32.858221 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.858201    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71ece96f-bf47-4f8f-9c40-49e873c87b3e-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr\" (UID: \"71ece96f-bf47-4f8f-9c40-49e873c87b3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"
Sep 17 22:45:32.864030 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.863990    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcw56\" (UniqueName: \"kubernetes.io/projected/71ece96f-bf47-4f8f-9c40-49e873c87b3e-kube-api-access-bcw56\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr\" (UID: \"71ece96f-bf47-4f8f-9c40-49e873c87b3e\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"
Sep 17 22:45:32.951517 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:32.951496    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"
Sep 17 22:45:33.009588 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:33.009554    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"]
Sep 17 22:45:33.013443 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:45:33.013410    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf438510f_56a1_47f8_9f7d_f6dc72b7d874.slice/crio-5e770905a5a9ad08015dc6685624472e84264f15c39a88d32bc7b0e83ad51aee WatchSource:0}: Error finding container 5e770905a5a9ad08015dc6685624472e84264f15c39a88d32bc7b0e83ad51aee: Status 404 returned error can't find the container with id 5e770905a5a9ad08015dc6685624472e84264f15c39a88d32bc7b0e83ad51aee
Sep 17 22:45:33.086254 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:33.086223    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" event={"ID":"f438510f-56a1-47f8-9f7d-f6dc72b7d874","Type":"ContainerStarted","Data":"5e770905a5a9ad08015dc6685624472e84264f15c39a88d32bc7b0e83ad51aee"}
Sep 17 22:45:33.221663 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:33.221615    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"]
Sep 17 22:45:33.230218 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:45:33.230190    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ece96f_bf47_4f8f_9c40_49e873c87b3e.slice/crio-f43af12a0631ab1d3c914f6dc1f20646ed77d45f460bcf3769d6e58c52da25a2 WatchSource:0}: Error finding container f43af12a0631ab1d3c914f6dc1f20646ed77d45f460bcf3769d6e58c52da25a2: Status 404 returned error can't find the container with id f43af12a0631ab1d3c914f6dc1f20646ed77d45f460bcf3769d6e58c52da25a2
Sep 17 22:45:33.581395 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:33.581286    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq" podUID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:45:34.091431 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.091389    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr" event={"ID":"71ece96f-bf47-4f8f-9c40-49e873c87b3e","Type":"ContainerStarted","Data":"81ed71f849ace2be94e2349a1bc073c6b5776a2f69ab861b55eb05e1b0ebe26a"}
Sep 17 22:45:34.091632 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.091475    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr" event={"ID":"71ece96f-bf47-4f8f-9c40-49e873c87b3e","Type":"ContainerStarted","Data":"f43af12a0631ab1d3c914f6dc1f20646ed77d45f460bcf3769d6e58c52da25a2"}
Sep 17 22:45:34.091632 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.091498    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"
Sep 17 22:45:34.092937 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.092912    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" event={"ID":"f438510f-56a1-47f8-9f7d-f6dc72b7d874","Type":"ContainerStarted","Data":"d37c37a6bf07000fec774e48038b7113f88f1bbb3e958015ef73790d2d68cdb3"}
Sep 17 22:45:34.106786 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.106742    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr" podStartSLOduration=2.106729799 podStartE2EDuration="2.106729799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:45:34.105259446 +0000 UTC m=+2811.114188990" watchObservedRunningTime="2026-09-17 22:45:34.106729799 +0000 UTC m=+2811.115659363"
Sep 17 22:45:34.393952 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.393928    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:45:34.473118 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.473085    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-workload-certs\") pod \"3a570df0-dc46-42da-ae11-c48773df5ff9\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") "
Sep 17 22:45:34.473276 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.473142    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-podinfo\") pod \"3a570df0-dc46-42da-ae11-c48773df5ff9\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") "
Sep 17 22:45:34.473276 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.473192    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gllj\" (UniqueName: \"kubernetes.io/projected/3a570df0-dc46-42da-ae11-c48773df5ff9-kube-api-access-8gllj\") pod \"3a570df0-dc46-42da-ae11-c48773df5ff9\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") "
Sep 17 22:45:34.473276 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.473228    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-token\") pod \"3a570df0-dc46-42da-ae11-c48773df5ff9\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") "
Sep 17 22:45:34.473433 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.473279    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-credential-socket\") pod \"3a570df0-dc46-42da-ae11-c48773df5ff9\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") "
Sep 17 22:45:34.473433 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.473316    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-workload-socket\") pod \"3a570df0-dc46-42da-ae11-c48773df5ff9\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") "
Sep 17 22:45:34.473433 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.473333    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a570df0-dc46-42da-ae11-c48773df5ff9" (UID: "3a570df0-dc46-42da-ae11-c48773df5ff9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:45:34.473433 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.473345    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-data\") pod \"3a570df0-dc46-42da-ae11-c48773df5ff9\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") "
Sep 17 22:45:34.473433 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.473405    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a570df0-dc46-42da-ae11-c48773df5ff9-istiod-ca-cert\") pod \"3a570df0-dc46-42da-ae11-c48773df5ff9\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") "
Sep 17 22:45:34.473682 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.473482    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-envoy\") pod \"3a570df0-dc46-42da-ae11-c48773df5ff9\" (UID: \"3a570df0-dc46-42da-ae11-c48773df5ff9\") "
Sep 17 22:45:34.473682 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.473510    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a570df0-dc46-42da-ae11-c48773df5ff9" (UID: "3a570df0-dc46-42da-ae11-c48773df5ff9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:45:34.473793 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.473728    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-data\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:34.473793 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.473745    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-workload-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:34.473898 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.473882    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a570df0-dc46-42da-ae11-c48773df5ff9" (UID: "3a570df0-dc46-42da-ae11-c48773df5ff9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:45:34.474182 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.474158    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a570df0-dc46-42da-ae11-c48773df5ff9" (UID: "3a570df0-dc46-42da-ae11-c48773df5ff9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:45:34.474446 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.474427    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a570df0-dc46-42da-ae11-c48773df5ff9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a570df0-dc46-42da-ae11-c48773df5ff9" (UID: "3a570df0-dc46-42da-ae11-c48773df5ff9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:45:34.475957 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.475932    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a570df0-dc46-42da-ae11-c48773df5ff9" (UID: "3a570df0-dc46-42da-ae11-c48773df5ff9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 22:45:34.476161 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.476140    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a570df0-dc46-42da-ae11-c48773df5ff9" (UID: "3a570df0-dc46-42da-ae11-c48773df5ff9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:45:34.476240 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.476154    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a570df0-dc46-42da-ae11-c48773df5ff9-kube-api-access-8gllj" (OuterVolumeSpecName: "kube-api-access-8gllj") pod "3a570df0-dc46-42da-ae11-c48773df5ff9" (UID: "3a570df0-dc46-42da-ae11-c48773df5ff9"). InnerVolumeSpecName "kube-api-access-8gllj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:45:34.477107 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.477075    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a570df0-dc46-42da-ae11-c48773df5ff9" (UID: "3a570df0-dc46-42da-ae11-c48773df5ff9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:45:34.575092 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.575055    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-podinfo\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:34.575092 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.575083    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gllj\" (UniqueName: \"kubernetes.io/projected/3a570df0-dc46-42da-ae11-c48773df5ff9-kube-api-access-8gllj\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:34.575092 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.575096    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-token\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:34.575322 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.575106    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-credential-socket\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:34.575322 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.575117    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-workload-socket\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:34.575322 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.575125    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a570df0-dc46-42da-ae11-c48773df5ff9-istiod-ca-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:34.575322 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:34.575133    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a570df0-dc46-42da-ae11-c48773df5ff9-istio-envoy\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:45:35.099991 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:35.099954    2567 generic.go:356] "Generic (PLEG): container finished" podID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerID="943087284abaad2fc1c199c50ce86ff9ff22c27626fb4f6f256bb6a02031d6cb" exitCode=0
Sep 17 22:45:35.100480 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:35.100019    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"
Sep 17 22:45:35.100480 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:35.100045    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq" event={"ID":"3a570df0-dc46-42da-ae11-c48773df5ff9","Type":"ContainerDied","Data":"943087284abaad2fc1c199c50ce86ff9ff22c27626fb4f6f256bb6a02031d6cb"}
Sep 17 22:45:35.100480 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:35.100097    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq" event={"ID":"3a570df0-dc46-42da-ae11-c48773df5ff9","Type":"ContainerDied","Data":"87a7e639979e86a8f1737bc7ce3e5dbab74a4fdd7e1623946192477cd9c72f78"}
Sep 17 22:45:35.100480 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:35.100119    2567 scope.go:117] "RemoveContainer" containerID="943087284abaad2fc1c199c50ce86ff9ff22c27626fb4f6f256bb6a02031d6cb"
Sep 17 22:45:35.108732 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:35.108715    2567 scope.go:117] "RemoveContainer" containerID="57b65790418f96689930601024bd83cc21efd56ca640a004b6b0cb201906eca5"
Sep 17 22:45:35.116408 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:35.116391    2567 scope.go:117] "RemoveContainer" containerID="943087284abaad2fc1c199c50ce86ff9ff22c27626fb4f6f256bb6a02031d6cb"
Sep 17 22:45:35.116644 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:45:35.116625    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943087284abaad2fc1c199c50ce86ff9ff22c27626fb4f6f256bb6a02031d6cb\": container with ID starting with 943087284abaad2fc1c199c50ce86ff9ff22c27626fb4f6f256bb6a02031d6cb not found: ID does not exist" containerID="943087284abaad2fc1c199c50ce86ff9ff22c27626fb4f6f256bb6a02031d6cb"
Sep 17 22:45:35.116699 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:35.116653    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943087284abaad2fc1c199c50ce86ff9ff22c27626fb4f6f256bb6a02031d6cb"} err="failed to get container status \"943087284abaad2fc1c199c50ce86ff9ff22c27626fb4f6f256bb6a02031d6cb\": rpc error: code = NotFound desc = could not find container \"943087284abaad2fc1c199c50ce86ff9ff22c27626fb4f6f256bb6a02031d6cb\": container with ID starting with 943087284abaad2fc1c199c50ce86ff9ff22c27626fb4f6f256bb6a02031d6cb not found: ID does not exist"
Sep 17 22:45:35.116699 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:35.116669    2567 scope.go:117] "RemoveContainer" containerID="57b65790418f96689930601024bd83cc21efd56ca640a004b6b0cb201906eca5"
Sep 17 22:45:35.116894 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:45:35.116876    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b65790418f96689930601024bd83cc21efd56ca640a004b6b0cb201906eca5\": container with ID starting with 57b65790418f96689930601024bd83cc21efd56ca640a004b6b0cb201906eca5 not found: ID does not exist" containerID="57b65790418f96689930601024bd83cc21efd56ca640a004b6b0cb201906eca5"
Sep 17 22:45:35.116944 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:35.116905    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b65790418f96689930601024bd83cc21efd56ca640a004b6b0cb201906eca5"} err="failed to get container status \"57b65790418f96689930601024bd83cc21efd56ca640a004b6b0cb201906eca5\": rpc error: code = NotFound desc = could not find container \"57b65790418f96689930601024bd83cc21efd56ca640a004b6b0cb201906eca5\": container with ID starting with 57b65790418f96689930601024bd83cc21efd56ca640a004b6b0cb201906eca5 not found: ID does not exist"
Sep 17 22:45:35.123122 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:35.123100    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"]
Sep 17 22:45:35.128123 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:35.128104    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d8488d74d-z9bzq"]
Sep 17 22:45:35.640006 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:35.639966    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a570df0-dc46-42da-ae11-c48773df5ff9" path="/var/lib/kubelet/pods/3a570df0-dc46-42da-ae11-c48773df5ff9/volumes"
Sep 17 22:45:36.105467 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:36.105434    2567 generic.go:356] "Generic (PLEG): container finished" podID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerID="d37c37a6bf07000fec774e48038b7113f88f1bbb3e958015ef73790d2d68cdb3" exitCode=0
Sep 17 22:45:36.105845 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:36.105505    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" event={"ID":"f438510f-56a1-47f8-9f7d-f6dc72b7d874","Type":"ContainerDied","Data":"d37c37a6bf07000fec774e48038b7113f88f1bbb3e958015ef73790d2d68cdb3"}
Sep 17 22:45:37.111353 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:37.111317    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" event={"ID":"f438510f-56a1-47f8-9f7d-f6dc72b7d874","Type":"ContainerStarted","Data":"d7da799fa53e137770f4aa89dc659c83114447691fd9fb93703f3723cfe0e15f"}
Sep 17 22:45:37.134277 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:37.134195    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" podStartSLOduration=2.984235649 podStartE2EDuration="5.134165115s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.149929466s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:45:37.131698918 +0000 UTC m=+2814.140628493" watchObservedRunningTime="2026-09-17 22:45:37.134165115 +0000 UTC m=+2814.143094656"
Sep 17 22:45:42.736530 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:42.736489    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:42.737046 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:42.736559    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:45:42.737941 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:42.737914    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 22:45:52.737593 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:45:52.737555    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 22:46:02.737654 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:46:02.737606    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 22:46:05.102086 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:46:05.102054    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"
Sep 17 22:46:12.737152 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:46:12.737114    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 22:46:22.737407 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:46:22.737358    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 22:46:32.737043 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:46:32.736974    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 22:46:42.736967 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:46:42.736920    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 22:46:52.737890 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:46:52.737832    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 22:47:02.737641 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:47:02.737598    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 22:47:12.737681 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:47:12.737639    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 22:47:22.746913 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:47:22.746880    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:47:22.754467 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:47:22.754441    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:47:54.730681 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:47:54.730646    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"]
Sep 17 22:47:54.734346 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:47:54.734319    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"]
Sep 17 22:47:55.102580 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:47:55.102493    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr" podUID="71ece96f-bf47-4f8f-9c40-49e873c87b3e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:48:05.102061 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:05.102024    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr" podUID="71ece96f-bf47-4f8f-9c40-49e873c87b3e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:48:09.750646 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:09.750551    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr" podUID="71ece96f-bf47-4f8f-9c40-49e873c87b3e" containerName="main" containerID="cri-o://81ed71f849ace2be94e2349a1bc073c6b5776a2f69ab861b55eb05e1b0ebe26a" gracePeriod=45
Sep 17 22:48:09.755974 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:09.755947    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="main" containerID="cri-o://d7da799fa53e137770f4aa89dc659c83114447691fd9fb93703f3723cfe0e15f" gracePeriod=45
Sep 17 22:48:10.419163 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.419137    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:48:10.455252 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.455227    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-tmp-dir\") pod \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") "
Sep 17 22:48:10.455392 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.455272    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-dshm\") pod \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") "
Sep 17 22:48:10.455392 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.455296    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-kserve-provision-location\") pod \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") "
Sep 17 22:48:10.455392 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.455328    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f438510f-56a1-47f8-9f7d-f6dc72b7d874-tls-certs\") pod \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") "
Sep 17 22:48:10.455392 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.455377    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blq2f\" (UniqueName: \"kubernetes.io/projected/f438510f-56a1-47f8-9f7d-f6dc72b7d874-kube-api-access-blq2f\") pod \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") "
Sep 17 22:48:10.455614 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.455404    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f438510f-56a1-47f8-9f7d-f6dc72b7d874-cabundle-cert\") pod \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") "
Sep 17 22:48:10.455614 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.455435    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-model-cache\") pod \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") "
Sep 17 22:48:10.455721 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.455658    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-model-cache" (OuterVolumeSpecName: "model-cache") pod "f438510f-56a1-47f8-9f7d-f6dc72b7d874" (UID: "f438510f-56a1-47f8-9f7d-f6dc72b7d874"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:10.456023 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.455973    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f438510f-56a1-47f8-9f7d-f6dc72b7d874-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f438510f-56a1-47f8-9f7d-f6dc72b7d874" (UID: "f438510f-56a1-47f8-9f7d-f6dc72b7d874"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:48:10.456666 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.456182    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-home\") pod \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\" (UID: \"f438510f-56a1-47f8-9f7d-f6dc72b7d874\") "
Sep 17 22:48:10.456666 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.456382    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f438510f-56a1-47f8-9f7d-f6dc72b7d874-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.456666 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.456399    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.457298 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.457258    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-home" (OuterVolumeSpecName: "home") pod "f438510f-56a1-47f8-9f7d-f6dc72b7d874" (UID: "f438510f-56a1-47f8-9f7d-f6dc72b7d874"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:10.458389 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.458356    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-dshm" (OuterVolumeSpecName: "dshm") pod "f438510f-56a1-47f8-9f7d-f6dc72b7d874" (UID: "f438510f-56a1-47f8-9f7d-f6dc72b7d874"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:10.458481 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.458383    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f438510f-56a1-47f8-9f7d-f6dc72b7d874-kube-api-access-blq2f" (OuterVolumeSpecName: "kube-api-access-blq2f") pod "f438510f-56a1-47f8-9f7d-f6dc72b7d874" (UID: "f438510f-56a1-47f8-9f7d-f6dc72b7d874"). InnerVolumeSpecName "kube-api-access-blq2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:48:10.458656 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.458629    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f438510f-56a1-47f8-9f7d-f6dc72b7d874-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f438510f-56a1-47f8-9f7d-f6dc72b7d874" (UID: "f438510f-56a1-47f8-9f7d-f6dc72b7d874"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:48:10.473348 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.473300    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f438510f-56a1-47f8-9f7d-f6dc72b7d874" (UID: "f438510f-56a1-47f8-9f7d-f6dc72b7d874"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:10.484453 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.484427    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f438510f-56a1-47f8-9f7d-f6dc72b7d874" (UID: "f438510f-56a1-47f8-9f7d-f6dc72b7d874"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:48:10.557705 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.557654    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.557705 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.557675    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.557705 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.557684    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.557705 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.557693    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f438510f-56a1-47f8-9f7d-f6dc72b7d874-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.557705 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.557701    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f438510f-56a1-47f8-9f7d-f6dc72b7d874-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.557920 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.557710    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blq2f\" (UniqueName: \"kubernetes.io/projected/f438510f-56a1-47f8-9f7d-f6dc72b7d874-kube-api-access-blq2f\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:48:10.703276 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.703239    2567 generic.go:356] "Generic (PLEG): container finished" podID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerID="d7da799fa53e137770f4aa89dc659c83114447691fd9fb93703f3723cfe0e15f" exitCode=0
Sep 17 22:48:10.703395 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.703333    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" event={"ID":"f438510f-56a1-47f8-9f7d-f6dc72b7d874","Type":"ContainerDied","Data":"d7da799fa53e137770f4aa89dc659c83114447691fd9fb93703f3723cfe0e15f"}
Sep 17 22:48:10.703395 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.703355    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"
Sep 17 22:48:10.703395 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.703387    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz" event={"ID":"f438510f-56a1-47f8-9f7d-f6dc72b7d874","Type":"ContainerDied","Data":"5e770905a5a9ad08015dc6685624472e84264f15c39a88d32bc7b0e83ad51aee"}
Sep 17 22:48:10.703501 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.703417    2567 scope.go:117] "RemoveContainer" containerID="d7da799fa53e137770f4aa89dc659c83114447691fd9fb93703f3723cfe0e15f"
Sep 17 22:48:10.711540 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.711524    2567 scope.go:117] "RemoveContainer" containerID="d37c37a6bf07000fec774e48038b7113f88f1bbb3e958015ef73790d2d68cdb3"
Sep 17 22:48:10.718530 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.718513    2567 scope.go:117] "RemoveContainer" containerID="d7da799fa53e137770f4aa89dc659c83114447691fd9fb93703f3723cfe0e15f"
Sep 17 22:48:10.718762 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:48:10.718747    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7da799fa53e137770f4aa89dc659c83114447691fd9fb93703f3723cfe0e15f\": container with ID starting with d7da799fa53e137770f4aa89dc659c83114447691fd9fb93703f3723cfe0e15f not found: ID does not exist" containerID="d7da799fa53e137770f4aa89dc659c83114447691fd9fb93703f3723cfe0e15f"
Sep 17 22:48:10.718841 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.718769    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7da799fa53e137770f4aa89dc659c83114447691fd9fb93703f3723cfe0e15f"} err="failed to get container status \"d7da799fa53e137770f4aa89dc659c83114447691fd9fb93703f3723cfe0e15f\": rpc error: code = NotFound desc = could not find container \"d7da799fa53e137770f4aa89dc659c83114447691fd9fb93703f3723cfe0e15f\": container with ID starting with d7da799fa53e137770f4aa89dc659c83114447691fd9fb93703f3723cfe0e15f not found: ID does not exist"
Sep 17 22:48:10.718841 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.718784    2567 scope.go:117] "RemoveContainer" containerID="d37c37a6bf07000fec774e48038b7113f88f1bbb3e958015ef73790d2d68cdb3"
Sep 17 22:48:10.719037 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:48:10.719011    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37c37a6bf07000fec774e48038b7113f88f1bbb3e958015ef73790d2d68cdb3\": container with ID starting with d37c37a6bf07000fec774e48038b7113f88f1bbb3e958015ef73790d2d68cdb3 not found: ID does not exist" containerID="d37c37a6bf07000fec774e48038b7113f88f1bbb3e958015ef73790d2d68cdb3"
Sep 17 22:48:10.719099 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.719042    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37c37a6bf07000fec774e48038b7113f88f1bbb3e958015ef73790d2d68cdb3"} err="failed to get container status \"d37c37a6bf07000fec774e48038b7113f88f1bbb3e958015ef73790d2d68cdb3\": rpc error: code = NotFound desc = could not find container \"d37c37a6bf07000fec774e48038b7113f88f1bbb3e958015ef73790d2d68cdb3\": container with ID starting with d37c37a6bf07000fec774e48038b7113f88f1bbb3e958015ef73790d2d68cdb3 not found: ID does not exist"
Sep 17 22:48:10.723966 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.723944    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"]
Sep 17 22:48:10.727692 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:10.727672    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-56bpmqz"]
Sep 17 22:48:11.639182 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:11.639150    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" path="/var/lib/kubelet/pods/f438510f-56a1-47f8-9f7d-f6dc72b7d874/volumes"
Sep 17 22:48:15.101742 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:15.101703    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr" podUID="71ece96f-bf47-4f8f-9c40-49e873c87b3e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:48:15.102221 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:15.101801    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"
Sep 17 22:48:25.102840 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:25.102800    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr" podUID="71ece96f-bf47-4f8f-9c40-49e873c87b3e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:48:35.101871 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:35.101832    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr" podUID="71ece96f-bf47-4f8f-9c40-49e873c87b3e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:48:39.983811 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:39.983788    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"
Sep 17 22:48:40.061634 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.061558    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71ece96f-bf47-4f8f-9c40-49e873c87b3e-tls-certs\") pod \"71ece96f-bf47-4f8f-9c40-49e873c87b3e\" (UID: \"71ece96f-bf47-4f8f-9c40-49e873c87b3e\") "
Sep 17 22:48:40.061634 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.061606    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcw56\" (UniqueName: \"kubernetes.io/projected/71ece96f-bf47-4f8f-9c40-49e873c87b3e-kube-api-access-bcw56\") pod \"71ece96f-bf47-4f8f-9c40-49e873c87b3e\" (UID: \"71ece96f-bf47-4f8f-9c40-49e873c87b3e\") "
Sep 17 22:48:40.063642 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.063618    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ece96f-bf47-4f8f-9c40-49e873c87b3e-kube-api-access-bcw56" (OuterVolumeSpecName: "kube-api-access-bcw56") pod "71ece96f-bf47-4f8f-9c40-49e873c87b3e" (UID: "71ece96f-bf47-4f8f-9c40-49e873c87b3e"). InnerVolumeSpecName "kube-api-access-bcw56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:48:40.063700 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.063623    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ece96f-bf47-4f8f-9c40-49e873c87b3e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71ece96f-bf47-4f8f-9c40-49e873c87b3e" (UID: "71ece96f-bf47-4f8f-9c40-49e873c87b3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:48:40.162238 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.162209    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71ece96f-bf47-4f8f-9c40-49e873c87b3e-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:48:40.162238 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.162234    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcw56\" (UniqueName: \"kubernetes.io/projected/71ece96f-bf47-4f8f-9c40-49e873c87b3e-kube-api-access-bcw56\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:48:40.806813 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.806776    2567 generic.go:356] "Generic (PLEG): container finished" podID="71ece96f-bf47-4f8f-9c40-49e873c87b3e" containerID="81ed71f849ace2be94e2349a1bc073c6b5776a2f69ab861b55eb05e1b0ebe26a" exitCode=0
Sep 17 22:48:40.806813 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.806814    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"
Sep 17 22:48:40.807031 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.806815    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr" event={"ID":"71ece96f-bf47-4f8f-9c40-49e873c87b3e","Type":"ContainerDied","Data":"81ed71f849ace2be94e2349a1bc073c6b5776a2f69ab861b55eb05e1b0ebe26a"}
Sep 17 22:48:40.807031 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.806944    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr" event={"ID":"71ece96f-bf47-4f8f-9c40-49e873c87b3e","Type":"ContainerDied","Data":"f43af12a0631ab1d3c914f6dc1f20646ed77d45f460bcf3769d6e58c52da25a2"}
Sep 17 22:48:40.807031 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.806967    2567 scope.go:117] "RemoveContainer" containerID="81ed71f849ace2be94e2349a1bc073c6b5776a2f69ab861b55eb05e1b0ebe26a"
Sep 17 22:48:40.816943 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.816914    2567 scope.go:117] "RemoveContainer" containerID="81ed71f849ace2be94e2349a1bc073c6b5776a2f69ab861b55eb05e1b0ebe26a"
Sep 17 22:48:40.817349 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:48:40.817316    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ed71f849ace2be94e2349a1bc073c6b5776a2f69ab861b55eb05e1b0ebe26a\": container with ID starting with 81ed71f849ace2be94e2349a1bc073c6b5776a2f69ab861b55eb05e1b0ebe26a not found: ID does not exist" containerID="81ed71f849ace2be94e2349a1bc073c6b5776a2f69ab861b55eb05e1b0ebe26a"
Sep 17 22:48:40.817438 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.817360    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ed71f849ace2be94e2349a1bc073c6b5776a2f69ab861b55eb05e1b0ebe26a"} err="failed to get container status \"81ed71f849ace2be94e2349a1bc073c6b5776a2f69ab861b55eb05e1b0ebe26a\": rpc error: code = NotFound desc = could not find container \"81ed71f849ace2be94e2349a1bc073c6b5776a2f69ab861b55eb05e1b0ebe26a\": container with ID starting with 81ed71f849ace2be94e2349a1bc073c6b5776a2f69ab861b55eb05e1b0ebe26a not found: ID does not exist"
Sep 17 22:48:40.831072 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.831052    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"]
Sep 17 22:48:40.836924 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:40.836905    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheph2pr"]
Sep 17 22:48:41.639244 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:41.639208    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ece96f-bf47-4f8f-9c40-49e873c87b3e" path="/var/lib/kubelet/pods/71ece96f-bf47-4f8f-9c40-49e873c87b3e/volumes"
Sep 17 22:48:57.564288 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564251    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"]
Sep 17 22:48:57.564674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564517    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="storage-initializer"
Sep 17 22:48:57.564674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564531    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="storage-initializer"
Sep 17 22:48:57.564674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564545    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="main"
Sep 17 22:48:57.564674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564550    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="main"
Sep 17 22:48:57.564674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564557    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerName="istio-proxy"
Sep 17 22:48:57.564674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564561    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerName="istio-proxy"
Sep 17 22:48:57.564674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564566    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ece96f-bf47-4f8f-9c40-49e873c87b3e" containerName="main"
Sep 17 22:48:57.564674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564572    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ece96f-bf47-4f8f-9c40-49e873c87b3e" containerName="main"
Sep 17 22:48:57.564674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564583    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerName="istio-proxy"
Sep 17 22:48:57.564674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564588    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerName="istio-proxy"
Sep 17 22:48:57.564674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564639    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerName="istio-proxy"
Sep 17 22:48:57.564674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564646    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a570df0-dc46-42da-ae11-c48773df5ff9" containerName="istio-proxy"
Sep 17 22:48:57.564674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564653    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ece96f-bf47-4f8f-9c40-49e873c87b3e" containerName="main"
Sep 17 22:48:57.564674 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.564658    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f438510f-56a1-47f8-9f7d-f6dc72b7d874" containerName="main"
Sep 17 22:48:57.566583 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.566560    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.569324 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.569302    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 22:48:57.569324 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.569309    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-k2cfp\""
Sep 17 22:48:57.569464 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.569355    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 22:48:57.569604 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.569588    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 22:48:57.570323 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.570300    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 22:48:57.570323 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.570311    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-6j9zp\""
Sep 17 22:48:57.570517 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.570298    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:48:57.582578 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.582557    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"]
Sep 17 22:48:57.683197 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.683171    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.683348 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.683202    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf278e34-366e-4875-b0a2-14c4103091f9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.683348 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.683221    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.683348 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.683238    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-dshm\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.683348 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.683306    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-home\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.683348 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.683335    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.683521 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.683372    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf278e34-366e-4875-b0a2-14c4103091f9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.683521 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.683426    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9nlv\" (UniqueName: \"kubernetes.io/projected/cf278e34-366e-4875-b0a2-14c4103091f9-kube-api-access-k9nlv\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.784419 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.784388    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf278e34-366e-4875-b0a2-14c4103091f9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.784517 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.784426    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9nlv\" (UniqueName: \"kubernetes.io/projected/cf278e34-366e-4875-b0a2-14c4103091f9-kube-api-access-k9nlv\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.784517 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.784458    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.784609 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.784557    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf278e34-366e-4875-b0a2-14c4103091f9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.784609 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.784580    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.784609 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.784598    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-dshm\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.784758 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.784614    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-home\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.784758 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.784635    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.784862 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.784829    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.784950 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.784928    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.784995 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.784954    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-home\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.785052 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.785019    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.785323 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.785302    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf278e34-366e-4875-b0a2-14c4103091f9-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.786819 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.786799    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-dshm\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.786946 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.786930    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf278e34-366e-4875-b0a2-14c4103091f9-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.794473 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.794451    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9nlv\" (UniqueName: \"kubernetes.io/projected/cf278e34-366e-4875-b0a2-14c4103091f9-kube-api-access-k9nlv\") pod \"custom-route-timeout-pd-test-kserve-7d486b5989-nctwq\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.797981 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.797959    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"]
Sep 17 22:48:57.800572 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.800558    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"
Sep 17 22:48:57.802942 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.802896    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-bs44t\""
Sep 17 22:48:57.811743 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.811721    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"]
Sep 17 22:48:57.875068 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.875015    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:57.886059 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.885990    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2jrh\" (UniqueName: \"kubernetes.io/projected/d26ef4e5-1859-43a3-875a-48c74ba3f399-kube-api-access-h2jrh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc\" (UID: \"d26ef4e5-1859-43a3-875a-48c74ba3f399\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"
Sep 17 22:48:57.886157 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.886074    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26ef4e5-1859-43a3-875a-48c74ba3f399-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc\" (UID: \"d26ef4e5-1859-43a3-875a-48c74ba3f399\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"
Sep 17 22:48:57.987254 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.987221    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2jrh\" (UniqueName: \"kubernetes.io/projected/d26ef4e5-1859-43a3-875a-48c74ba3f399-kube-api-access-h2jrh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc\" (UID: \"d26ef4e5-1859-43a3-875a-48c74ba3f399\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"
Sep 17 22:48:57.987387 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.987273    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26ef4e5-1859-43a3-875a-48c74ba3f399-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc\" (UID: \"d26ef4e5-1859-43a3-875a-48c74ba3f399\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"
Sep 17 22:48:57.989516 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.989490    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26ef4e5-1859-43a3-875a-48c74ba3f399-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc\" (UID: \"d26ef4e5-1859-43a3-875a-48c74ba3f399\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"
Sep 17 22:48:57.995472 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:57.995450    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2jrh\" (UniqueName: \"kubernetes.io/projected/d26ef4e5-1859-43a3-875a-48c74ba3f399-kube-api-access-h2jrh\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc\" (UID: \"d26ef4e5-1859-43a3-875a-48c74ba3f399\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"
Sep 17 22:48:58.112119 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:58.112087    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"
Sep 17 22:48:58.150607 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:58.150583    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"]
Sep 17 22:48:58.152984 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:48:58.152960    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf278e34_366e_4875_b0a2_14c4103091f9.slice/crio-44dc1dbb1bcb450c19c837c876975228acdcef811f3c1734189a1c0d6b19f08c WatchSource:0}: Error finding container 44dc1dbb1bcb450c19c837c876975228acdcef811f3c1734189a1c0d6b19f08c: Status 404 returned error can't find the container with id 44dc1dbb1bcb450c19c837c876975228acdcef811f3c1734189a1c0d6b19f08c
Sep 17 22:48:58.380741 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:58.380529    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"]
Sep 17 22:48:58.383217 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:48:58.383174    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26ef4e5_1859_43a3_875a_48c74ba3f399.slice/crio-3f65bbd62feda6ec23a4aed405e3341aed2560d3edf8c5041ef14b6d70788c59 WatchSource:0}: Error finding container 3f65bbd62feda6ec23a4aed405e3341aed2560d3edf8c5041ef14b6d70788c59: Status 404 returned error can't find the container with id 3f65bbd62feda6ec23a4aed405e3341aed2560d3edf8c5041ef14b6d70788c59
Sep 17 22:48:58.874575 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:58.874537    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc" event={"ID":"d26ef4e5-1859-43a3-875a-48c74ba3f399","Type":"ContainerStarted","Data":"b32a872f82166eace66a7159d414f64775aa79dc10d1217f17a63a04360343e8"}
Sep 17 22:48:58.875045 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:58.874659    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"
Sep 17 22:48:58.875045 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:58.874685    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc" event={"ID":"d26ef4e5-1859-43a3-875a-48c74ba3f399","Type":"ContainerStarted","Data":"3f65bbd62feda6ec23a4aed405e3341aed2560d3edf8c5041ef14b6d70788c59"}
Sep 17 22:48:58.875847 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:58.875826    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" event={"ID":"cf278e34-366e-4875-b0a2-14c4103091f9","Type":"ContainerStarted","Data":"dbcf97014fe43f9bd7d9cd4b7fe7299a13627db7e4e73594cf639f9138869ac3"}
Sep 17 22:48:58.875930 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:58.875862    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" event={"ID":"cf278e34-366e-4875-b0a2-14c4103091f9","Type":"ContainerStarted","Data":"44dc1dbb1bcb450c19c837c876975228acdcef811f3c1734189a1c0d6b19f08c"}
Sep 17 22:48:58.875930 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:58.875900    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:48:58.892098 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:58.892051    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc" podStartSLOduration=1.892031756 podStartE2EDuration="1.892031756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:48:58.890774472 +0000 UTC m=+3015.899704013" watchObservedRunningTime="2026-09-17 22:48:58.892031756 +0000 UTC m=+3015.900961304"
Sep 17 22:48:59.881187 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:48:59.881147    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" event={"ID":"cf278e34-366e-4875-b0a2-14c4103091f9","Type":"ContainerStarted","Data":"28337cd77ba52855c2a0745a9b28b0d06138a05f23196733f4956efb416ab5c3"}
Sep 17 22:49:01.891038 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:49:01.890989    2567 generic.go:356] "Generic (PLEG): container finished" podID="cf278e34-366e-4875-b0a2-14c4103091f9" containerID="28337cd77ba52855c2a0745a9b28b0d06138a05f23196733f4956efb416ab5c3" exitCode=0
Sep 17 22:49:01.891038 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:49:01.891041    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" event={"ID":"cf278e34-366e-4875-b0a2-14c4103091f9","Type":"ContainerDied","Data":"28337cd77ba52855c2a0745a9b28b0d06138a05f23196733f4956efb416ab5c3"}
Sep 17 22:49:02.897391 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:49:02.897349    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" event={"ID":"cf278e34-366e-4875-b0a2-14c4103091f9","Type":"ContainerStarted","Data":"9b68692214cc19ebd88afc36e5b856904a95d5f1bf630618a5f22d0fba6006b0"}
Sep 17 22:49:02.920684 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:49:02.920629    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" podStartSLOduration=3.887913561 podStartE2EDuration="5.920614018s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.032700457s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:49:02.919711826 +0000 UTC m=+3019.928641369" watchObservedRunningTime="2026-09-17 22:49:02.920614018 +0000 UTC m=+3019.929543558"
Sep 17 22:49:07.875308 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:49:07.875275    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:49:07.875730 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:49:07.875329    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:49:07.876631 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:49:07.876604    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Sep 17 22:49:17.876168 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:49:17.876111    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Sep 17 22:49:17.888883 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:49:17.888851    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:49:27.875677 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:49:27.875613    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Sep 17 22:49:29.882612 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:49:29.882575    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"
Sep 17 22:49:37.876325 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:49:37.876227    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Sep 17 22:49:47.876108 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:49:47.876056    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Sep 17 22:49:57.876158 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:49:57.876108    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Sep 17 22:50:07.875430 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:50:07.875383    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Sep 17 22:50:17.875739 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:50:17.875671    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Sep 17 22:50:27.876081 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:50:27.876030    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Sep 17 22:50:37.875888 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:50:37.875831    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8001/health\": dial tcp 10.133.0.53:8001: connect: connection refused"
Sep 17 22:50:47.885446 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:50:47.885413    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:50:48.319679 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:50:48.319649    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:51:37.780299 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:37.780106    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"]
Sep 17 22:51:37.781651 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:37.781621    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"]
Sep 17 22:51:39.882823 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:39.882784    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc" podUID="d26ef4e5-1859-43a3-875a-48c74ba3f399" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:51:49.882846 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:49.882807    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc" podUID="d26ef4e5-1859-43a3-875a-48c74ba3f399" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:51:52.800150 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:52.800098    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc" podUID="d26ef4e5-1859-43a3-875a-48c74ba3f399" containerName="main" containerID="cri-o://b32a872f82166eace66a7159d414f64775aa79dc10d1217f17a63a04360343e8" gracePeriod=45
Sep 17 22:51:52.803588 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:52.803554    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="main" containerID="cri-o://9b68692214cc19ebd88afc36e5b856904a95d5f1bf630618a5f22d0fba6006b0" gracePeriod=45
Sep 17 22:51:53.317840 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.317785    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="llm-d-routing-sidecar" containerID="cri-o://dbcf97014fe43f9bd7d9cd4b7fe7299a13627db7e4e73594cf639f9138869ac3" gracePeriod=45
Sep 17 22:51:53.544303 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.544233    2567 generic.go:356] "Generic (PLEG): container finished" podID="cf278e34-366e-4875-b0a2-14c4103091f9" containerID="9b68692214cc19ebd88afc36e5b856904a95d5f1bf630618a5f22d0fba6006b0" exitCode=0
Sep 17 22:51:53.544303 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.544259    2567 generic.go:356] "Generic (PLEG): container finished" podID="cf278e34-366e-4875-b0a2-14c4103091f9" containerID="dbcf97014fe43f9bd7d9cd4b7fe7299a13627db7e4e73594cf639f9138869ac3" exitCode=0
Sep 17 22:51:53.544464 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.544307    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" event={"ID":"cf278e34-366e-4875-b0a2-14c4103091f9","Type":"ContainerDied","Data":"9b68692214cc19ebd88afc36e5b856904a95d5f1bf630618a5f22d0fba6006b0"}
Sep 17 22:51:53.544464 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.544374    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" event={"ID":"cf278e34-366e-4875-b0a2-14c4103091f9","Type":"ContainerDied","Data":"dbcf97014fe43f9bd7d9cd4b7fe7299a13627db7e4e73594cf639f9138869ac3"}
Sep 17 22:51:53.544464 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.544397    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq" event={"ID":"cf278e34-366e-4875-b0a2-14c4103091f9","Type":"ContainerDied","Data":"44dc1dbb1bcb450c19c837c876975228acdcef811f3c1734189a1c0d6b19f08c"}
Sep 17 22:51:53.544464 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.544419    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44dc1dbb1bcb450c19c837c876975228acdcef811f3c1734189a1c0d6b19f08c"
Sep 17 22:51:53.550498 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.550481    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:51:53.729963 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.729942    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-kserve-provision-location\") pod \"cf278e34-366e-4875-b0a2-14c4103091f9\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") "
Sep 17 22:51:53.730106 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.729969    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf278e34-366e-4875-b0a2-14c4103091f9-tls-certs\") pod \"cf278e34-366e-4875-b0a2-14c4103091f9\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") "
Sep 17 22:51:53.730106 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.729988    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-dshm\") pod \"cf278e34-366e-4875-b0a2-14c4103091f9\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") "
Sep 17 22:51:53.730106 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.730018    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-home\") pod \"cf278e34-366e-4875-b0a2-14c4103091f9\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") "
Sep 17 22:51:53.730106 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.730044    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-tmp-dir\") pod \"cf278e34-366e-4875-b0a2-14c4103091f9\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") "
Sep 17 22:51:53.730106 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.730092    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-model-cache\") pod \"cf278e34-366e-4875-b0a2-14c4103091f9\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") "
Sep 17 22:51:53.730311 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.730128    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9nlv\" (UniqueName: \"kubernetes.io/projected/cf278e34-366e-4875-b0a2-14c4103091f9-kube-api-access-k9nlv\") pod \"cf278e34-366e-4875-b0a2-14c4103091f9\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") "
Sep 17 22:51:53.730311 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.730186    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf278e34-366e-4875-b0a2-14c4103091f9-cabundle-cert\") pod \"cf278e34-366e-4875-b0a2-14c4103091f9\" (UID: \"cf278e34-366e-4875-b0a2-14c4103091f9\") "
Sep 17 22:51:53.730484 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.730458    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf278e34-366e-4875-b0a2-14c4103091f9" (UID: "cf278e34-366e-4875-b0a2-14c4103091f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:51:53.730696 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.730673    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf278e34-366e-4875-b0a2-14c4103091f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf278e34-366e-4875-b0a2-14c4103091f9" (UID: "cf278e34-366e-4875-b0a2-14c4103091f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:51:53.730783 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.730751    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-home" (OuterVolumeSpecName: "home") pod "cf278e34-366e-4875-b0a2-14c4103091f9" (UID: "cf278e34-366e-4875-b0a2-14c4103091f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:51:53.732406 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.732374    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf278e34-366e-4875-b0a2-14c4103091f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf278e34-366e-4875-b0a2-14c4103091f9" (UID: "cf278e34-366e-4875-b0a2-14c4103091f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:51:53.732406 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.732380    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf278e34-366e-4875-b0a2-14c4103091f9-kube-api-access-k9nlv" (OuterVolumeSpecName: "kube-api-access-k9nlv") pod "cf278e34-366e-4875-b0a2-14c4103091f9" (UID: "cf278e34-366e-4875-b0a2-14c4103091f9"). InnerVolumeSpecName "kube-api-access-k9nlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:51:53.732543 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.732447    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-dshm" (OuterVolumeSpecName: "dshm") pod "cf278e34-366e-4875-b0a2-14c4103091f9" (UID: "cf278e34-366e-4875-b0a2-14c4103091f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:51:53.742827 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.742806    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf278e34-366e-4875-b0a2-14c4103091f9" (UID: "cf278e34-366e-4875-b0a2-14c4103091f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:51:53.753510 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.753488    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf278e34-366e-4875-b0a2-14c4103091f9" (UID: "cf278e34-366e-4875-b0a2-14c4103091f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:51:53.831609 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.831564    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:51:53.831609 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.831584    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf278e34-366e-4875-b0a2-14c4103091f9-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:51:53.831609 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.831592    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:51:53.831609 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.831599    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:51:53.831609 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.831607    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:51:53.832057 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.831614    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf278e34-366e-4875-b0a2-14c4103091f9-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:51:53.832057 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.831621    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9nlv\" (UniqueName: \"kubernetes.io/projected/cf278e34-366e-4875-b0a2-14c4103091f9-kube-api-access-k9nlv\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:51:53.832057 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:53.831629    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf278e34-366e-4875-b0a2-14c4103091f9-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:51:54.547322 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:54.547287    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"
Sep 17 22:51:54.567650 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:54.567617    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"]
Sep 17 22:51:54.571485 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:54.571458    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-7d486b5989-nctwq"]
Sep 17 22:51:55.640090 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:55.640057    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" path="/var/lib/kubelet/pods/cf278e34-366e-4875-b0a2-14c4103091f9/volumes"
Sep 17 22:51:59.882881 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:59.882843    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc" podUID="d26ef4e5-1859-43a3-875a-48c74ba3f399" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:51:59.883282 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:51:59.882949    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"
Sep 17 22:52:09.882845 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:09.882807    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc" podUID="d26ef4e5-1859-43a3-875a-48c74ba3f399" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:52:19.882989 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:19.882952    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc" podUID="d26ef4e5-1859-43a3-875a-48c74ba3f399" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:52:23.033488 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.033465    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"
Sep 17 22:52:23.124880 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.124812    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26ef4e5-1859-43a3-875a-48c74ba3f399-tls-certs\") pod \"d26ef4e5-1859-43a3-875a-48c74ba3f399\" (UID: \"d26ef4e5-1859-43a3-875a-48c74ba3f399\") "
Sep 17 22:52:23.124880 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.124868    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2jrh\" (UniqueName: \"kubernetes.io/projected/d26ef4e5-1859-43a3-875a-48c74ba3f399-kube-api-access-h2jrh\") pod \"d26ef4e5-1859-43a3-875a-48c74ba3f399\" (UID: \"d26ef4e5-1859-43a3-875a-48c74ba3f399\") "
Sep 17 22:52:23.126841 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.126814    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26ef4e5-1859-43a3-875a-48c74ba3f399-kube-api-access-h2jrh" (OuterVolumeSpecName: "kube-api-access-h2jrh") pod "d26ef4e5-1859-43a3-875a-48c74ba3f399" (UID: "d26ef4e5-1859-43a3-875a-48c74ba3f399"). InnerVolumeSpecName "kube-api-access-h2jrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:52:23.126946 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.126845    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26ef4e5-1859-43a3-875a-48c74ba3f399-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d26ef4e5-1859-43a3-875a-48c74ba3f399" (UID: "d26ef4e5-1859-43a3-875a-48c74ba3f399"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:52:23.225491 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.225463    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2jrh\" (UniqueName: \"kubernetes.io/projected/d26ef4e5-1859-43a3-875a-48c74ba3f399-kube-api-access-h2jrh\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:52:23.225491 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.225487    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26ef4e5-1859-43a3-875a-48c74ba3f399-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:52:23.658114 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.658083    2567 generic.go:356] "Generic (PLEG): container finished" podID="d26ef4e5-1859-43a3-875a-48c74ba3f399" containerID="b32a872f82166eace66a7159d414f64775aa79dc10d1217f17a63a04360343e8" exitCode=0
Sep 17 22:52:23.658223 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.658118    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"
Sep 17 22:52:23.658223 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.658163    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc" event={"ID":"d26ef4e5-1859-43a3-875a-48c74ba3f399","Type":"ContainerDied","Data":"b32a872f82166eace66a7159d414f64775aa79dc10d1217f17a63a04360343e8"}
Sep 17 22:52:23.658223 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.658202    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc" event={"ID":"d26ef4e5-1859-43a3-875a-48c74ba3f399","Type":"ContainerDied","Data":"3f65bbd62feda6ec23a4aed405e3341aed2560d3edf8c5041ef14b6d70788c59"}
Sep 17 22:52:23.658325 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.658224    2567 scope.go:117] "RemoveContainer" containerID="b32a872f82166eace66a7159d414f64775aa79dc10d1217f17a63a04360343e8"
Sep 17 22:52:23.666391 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.666373    2567 scope.go:117] "RemoveContainer" containerID="b32a872f82166eace66a7159d414f64775aa79dc10d1217f17a63a04360343e8"
Sep 17 22:52:23.666677 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:52:23.666658    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b32a872f82166eace66a7159d414f64775aa79dc10d1217f17a63a04360343e8\": container with ID starting with b32a872f82166eace66a7159d414f64775aa79dc10d1217f17a63a04360343e8 not found: ID does not exist" containerID="b32a872f82166eace66a7159d414f64775aa79dc10d1217f17a63a04360343e8"
Sep 17 22:52:23.666733 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.666690    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32a872f82166eace66a7159d414f64775aa79dc10d1217f17a63a04360343e8"} err="failed to get container status \"b32a872f82166eace66a7159d414f64775aa79dc10d1217f17a63a04360343e8\": rpc error: code = NotFound desc = could not find container \"b32a872f82166eace66a7159d414f64775aa79dc10d1217f17a63a04360343e8\": container with ID starting with b32a872f82166eace66a7159d414f64775aa79dc10d1217f17a63a04360343e8 not found: ID does not exist"
Sep 17 22:52:23.674601 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.674582    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"]
Sep 17 22:52:23.678407 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:23.678387    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48649zc"]
Sep 17 22:52:25.640176 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:25.640146    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d26ef4e5-1859-43a3-875a-48c74ba3f399" path="/var/lib/kubelet/pods/d26ef4e5-1859-43a3-875a-48c74ba3f399/volumes"
Sep 17 22:52:28.310223 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.310191    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"]
Sep 17 22:52:28.310590 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.310443    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="llm-d-routing-sidecar"
Sep 17 22:52:28.310590 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.310454    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="llm-d-routing-sidecar"
Sep 17 22:52:28.310590 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.310467    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d26ef4e5-1859-43a3-875a-48c74ba3f399" containerName="main"
Sep 17 22:52:28.310590 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.310474    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d26ef4e5-1859-43a3-875a-48c74ba3f399" containerName="main"
Sep 17 22:52:28.310590 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.310480    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="storage-initializer"
Sep 17 22:52:28.310590 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.310486    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="storage-initializer"
Sep 17 22:52:28.310590 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.310491    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="main"
Sep 17 22:52:28.310590 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.310496    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="main"
Sep 17 22:52:28.310590 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.310543    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="main"
Sep 17 22:52:28.310590 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.310550    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d26ef4e5-1859-43a3-875a-48c74ba3f399" containerName="main"
Sep 17 22:52:28.310590 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.310557    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf278e34-366e-4875-b0a2-14c4103091f9" containerName="llm-d-routing-sidecar"
Sep 17 22:52:28.313289 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.313252    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.315526 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.315501    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 22:52:28.315526 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.315518    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 22:52:28.315696 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.315545    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-m77m2\""
Sep 17 22:52:28.315696 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.315549    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 17 22:52:28.326740 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.326711    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"]
Sep 17 22:52:28.357416 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.357386    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca931302-608d-486f-9984-c0ea421f31e7-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.357517 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.357419    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-istio-data\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.357517 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.357492    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-workload-certs\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.357517 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.357509    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca931302-608d-486f-9984-c0ea421f31e7-istio-token\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.357630 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.357531    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-istio-envoy\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.357630 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.357556    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-credential-socket\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.357630 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.357576    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rtsx\" (UniqueName: \"kubernetes.io/projected/ca931302-608d-486f-9984-c0ea421f31e7-kube-api-access-6rtsx\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.357630 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.357618    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-workload-socket\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.357755 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.357635    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca931302-608d-486f-9984-c0ea421f31e7-istio-podinfo\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.458369 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.458346    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca931302-608d-486f-9984-c0ea421f31e7-istio-token\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.458464 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.458371    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-istio-envoy\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.458464 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.458389    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-credential-socket\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.458464 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.458404    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rtsx\" (UniqueName: \"kubernetes.io/projected/ca931302-608d-486f-9984-c0ea421f31e7-kube-api-access-6rtsx\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.458464 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.458429    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-workload-socket\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.458642 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.458576    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca931302-608d-486f-9984-c0ea421f31e7-istio-podinfo\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.458642 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.458620    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca931302-608d-486f-9984-c0ea421f31e7-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.458748 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.458651    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-istio-data\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.458748 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.458740    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-workload-certs\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.458850 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.458810    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-credential-socket\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.458907 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.458860    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-workload-socket\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.459043 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.459022    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-istio-data\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.459117 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.459071    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-workload-certs\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.459266 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.459246    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca931302-608d-486f-9984-c0ea421f31e7-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.460897 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.460880    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-istio-envoy\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.461056 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.461038    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca931302-608d-486f-9984-c0ea421f31e7-istio-podinfo\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.465892 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.465868    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca931302-608d-486f-9984-c0ea421f31e7-istio-token\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.466296 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.466275    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rtsx\" (UniqueName: \"kubernetes.io/projected/ca931302-608d-486f-9984-c0ea421f31e7-kube-api-access-6rtsx\") pod \"router-gateway-2-openshift-default-77d9dd58dd-8mtvg\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.624110 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.624048    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:28.899188 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.899152    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"]
Sep 17 22:52:28.901932 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:52:28.901902    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca931302_608d_486f_9984_c0ea421f31e7.slice/crio-12dbcee93d8312f387ab1c508a226079e89aef78d82beed4817734624b421673 WatchSource:0}: Error finding container 12dbcee93d8312f387ab1c508a226079e89aef78d82beed4817734624b421673: Status 404 returned error can't find the container with id 12dbcee93d8312f387ab1c508a226079e89aef78d82beed4817734624b421673
Sep 17 22:52:28.903933 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.903900    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:52:28.904067 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.903982    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:52:28.904067 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:28.904026    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:52:29.683591 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:29.683561    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg" event={"ID":"ca931302-608d-486f-9984-c0ea421f31e7","Type":"ContainerStarted","Data":"311b772836304d98bc55d631cec03adcc7de67454a93d112c993a6bac4148f18"}
Sep 17 22:52:29.684030 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:29.683604    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg" event={"ID":"ca931302-608d-486f-9984-c0ea421f31e7","Type":"ContainerStarted","Data":"12dbcee93d8312f387ab1c508a226079e89aef78d82beed4817734624b421673"}
Sep 17 22:52:29.704444 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:29.704402    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg" podStartSLOduration=1.704389393 podStartE2EDuration="1.704389393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:52:29.703062681 +0000 UTC m=+3226.711992222" watchObservedRunningTime="2026-09-17 22:52:29.704389393 +0000 UTC m=+3226.713318955"
Sep 17 22:52:30.624712 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:30.624680    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:30.629873 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:30.629847    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:30.687315 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:30.687293    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:30.688087 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:30.688067    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:39.377245 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.377199    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"]
Sep 17 22:52:39.381323 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.381302    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.383848 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.383817    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-xs6r6\""
Sep 17 22:52:39.384037 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.383868    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 22:52:39.384037 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.383944    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-hxsc5\""
Sep 17 22:52:39.384037 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.383946    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 22:52:39.384513 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.384496    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:52:39.392266 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.392245    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"]
Sep 17 22:52:39.444867 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.444789    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-dshm\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.444867 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.444825    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp6m7\" (UniqueName: \"kubernetes.io/projected/19624b14-4973-4b94-97d3-3c0572dcee03-kube-api-access-vp6m7\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.444867 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.444862    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-home\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.467405 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.444883    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-model-cache\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.467405 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.444906    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.467405 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.444923    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.467405 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.444958    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19624b14-4973-4b94-97d3-3c0572dcee03-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.467405 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.445069    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19624b14-4973-4b94-97d3-3c0572dcee03-tls-certs\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.545920 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.545890    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19624b14-4973-4b94-97d3-3c0572dcee03-tls-certs\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.546123 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.545946    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-dshm\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.546123 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.545983    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6m7\" (UniqueName: \"kubernetes.io/projected/19624b14-4973-4b94-97d3-3c0572dcee03-kube-api-access-vp6m7\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.546253 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.546141    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-home\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.546253 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.546181    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-model-cache\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.546253 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.546220    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.546429 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.546253    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.546429 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.546281    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19624b14-4973-4b94-97d3-3c0572dcee03-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.546565 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.546538    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-home\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.546736 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.546585    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.546847 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.546643    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.546927 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.546840    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-model-cache\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.547194 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.547173    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19624b14-4973-4b94-97d3-3c0572dcee03-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.548309 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.548282    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-dshm\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.548532 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.548513    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19624b14-4973-4b94-97d3-3c0572dcee03-tls-certs\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.553900 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.553873    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp6m7\" (UniqueName: \"kubernetes.io/projected/19624b14-4973-4b94-97d3-3c0572dcee03-kube-api-access-vp6m7\") pod \"router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.691698 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.691666    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:39.971618 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:39.971586    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"]
Sep 17 22:52:39.973970 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:52:39.973940    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19624b14_4973_4b94_97d3_3c0572dcee03.slice/crio-45ec8451f65ed35cd599a8111f40288c4a670e94831a60cf356f9678888537db WatchSource:0}: Error finding container 45ec8451f65ed35cd599a8111f40288c4a670e94831a60cf356f9678888537db: Status 404 returned error can't find the container with id 45ec8451f65ed35cd599a8111f40288c4a670e94831a60cf356f9678888537db
Sep 17 22:52:40.724616 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:40.724584    2567 generic.go:356] "Generic (PLEG): container finished" podID="ca931302-608d-486f-9984-c0ea421f31e7" containerID="311b772836304d98bc55d631cec03adcc7de67454a93d112c993a6bac4148f18" exitCode=0
Sep 17 22:52:40.725085 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:40.724655    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg" event={"ID":"ca931302-608d-486f-9984-c0ea421f31e7","Type":"ContainerDied","Data":"311b772836304d98bc55d631cec03adcc7de67454a93d112c993a6bac4148f18"}
Sep 17 22:52:40.725085 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:40.725062    2567 scope.go:117] "RemoveContainer" containerID="311b772836304d98bc55d631cec03adcc7de67454a93d112c993a6bac4148f18"
Sep 17 22:52:40.726057 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:40.726020    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:52:40.726178 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:40.726108    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:52:40.726178 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:40.726161    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 22:52:40.726388 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:40.726369    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" event={"ID":"19624b14-4973-4b94-97d3-3c0572dcee03","Type":"ContainerStarted","Data":"fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4"}
Sep 17 22:52:40.726465 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:40.726408    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" event={"ID":"19624b14-4973-4b94-97d3-3c0572dcee03","Type":"ContainerStarted","Data":"45ec8451f65ed35cd599a8111f40288c4a670e94831a60cf356f9678888537db"}
Sep 17 22:52:40.726465 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:40.726433    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:41.731952 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:41.731918    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" event={"ID":"19624b14-4973-4b94-97d3-3c0572dcee03","Type":"ContainerStarted","Data":"91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615"}
Sep 17 22:52:41.734031 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:41.733987    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg" event={"ID":"ca931302-608d-486f-9984-c0ea421f31e7","Type":"ContainerStarted","Data":"2181a64f7465116394dafaa712df2e010c558559cef69da0c7215bfd532d2ff0"}
Sep 17 22:52:42.624384 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:42.624348    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:42.625219 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:42.625186    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg" podUID="ca931302-608d-486f-9984-c0ea421f31e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 17 22:52:43.625429 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:43.625392    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg" podUID="ca931302-608d-486f-9984-c0ea421f31e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 17 22:52:43.757773 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:43.757604    2567 generic.go:356] "Generic (PLEG): container finished" podID="19624b14-4973-4b94-97d3-3c0572dcee03" containerID="91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615" exitCode=0
Sep 17 22:52:43.757773 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:43.757679    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" event={"ID":"19624b14-4973-4b94-97d3-3c0572dcee03","Type":"ContainerDied","Data":"91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615"}
Sep 17 22:52:44.625087 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:44.625047    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg" podUID="ca931302-608d-486f-9984-c0ea421f31e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 17 22:52:44.765507 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:44.765460    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" event={"ID":"19624b14-4973-4b94-97d3-3c0572dcee03","Type":"ContainerStarted","Data":"be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb"}
Sep 17 22:52:44.791800 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:44.791725    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" podStartSLOduration=3.3855564129999998 podStartE2EDuration="5.791705224s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.406148811s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:52:44.789672074 +0000 UTC m=+3241.798601630" watchObservedRunningTime="2026-09-17 22:52:44.791705224 +0000 UTC m=+3241.800634765"
Sep 17 22:52:45.624609 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:45.624568    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg" podUID="ca931302-608d-486f-9984-c0ea421f31e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 17 22:52:45.688243 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:45.688211    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:46.624981 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:46.624941    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg" podUID="ca931302-608d-486f-9984-c0ea421f31e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 17 22:52:47.629148 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:47.629116    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:47.630193 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:47.630174    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:52:49.692066 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:49.692037    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:49.692539 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:49.692094    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:52:49.693388 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:49.693356    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8001/health\": dial tcp 10.133.0.56:8001: connect: connection refused"
Sep 17 22:52:59.692389 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:59.692333    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8001/health\": dial tcp 10.133.0.56:8001: connect: connection refused"
Sep 17 22:52:59.703920 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:52:59.703898    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:53:09.693088 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:53:09.693039    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8001/health\": dial tcp 10.133.0.56:8001: connect: connection refused"
Sep 17 22:53:19.693183 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:53:19.693132    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8001/health\": dial tcp 10.133.0.56:8001: connect: connection refused"
Sep 17 22:53:29.692679 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:53:29.692628    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8001/health\": dial tcp 10.133.0.56:8001: connect: connection refused"
Sep 17 22:53:39.692867 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:53:39.692756    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8001/health\": dial tcp 10.133.0.56:8001: connect: connection refused"
Sep 17 22:53:49.692349 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:53:49.692297    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8001/health\": dial tcp 10.133.0.56:8001: connect: connection refused"
Sep 17 22:53:59.692495 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:53:59.692445    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8001/health\": dial tcp 10.133.0.56:8001: connect: connection refused"
Sep 17 22:54:09.692700 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:54:09.692601    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8001/health\": dial tcp 10.133.0.56:8001: connect: connection refused"
Sep 17 22:54:19.692499 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:54:19.692450    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8001/health\": dial tcp 10.133.0.56:8001: connect: connection refused"
Sep 17 22:54:29.702141 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:54:29.702110    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:54:29.713822 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:54:29.713798    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:54:48.343740 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:54:48.343709    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"]
Sep 17 22:55:03.365359 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:03.365303    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="main" containerID="cri-o://be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb" gracePeriod=45
Sep 17 22:55:03.982160 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:03.982125    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="llm-d-routing-sidecar" containerID="cri-o://fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4" gracePeriod=45
Sep 17 22:55:04.219771 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.219747    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:55:04.282985 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.282935    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19624b14-4973-4b94-97d3-3c0572dcee03-cabundle-cert\") pod \"19624b14-4973-4b94-97d3-3c0572dcee03\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") "
Sep 17 22:55:04.283194 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.283032    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-model-cache\") pod \"19624b14-4973-4b94-97d3-3c0572dcee03\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") "
Sep 17 22:55:04.283194 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.283055    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-dshm\") pod \"19624b14-4973-4b94-97d3-3c0572dcee03\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") "
Sep 17 22:55:04.283194 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.283075    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-home\") pod \"19624b14-4973-4b94-97d3-3c0572dcee03\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") "
Sep 17 22:55:04.283194 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.283110    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp6m7\" (UniqueName: \"kubernetes.io/projected/19624b14-4973-4b94-97d3-3c0572dcee03-kube-api-access-vp6m7\") pod \"19624b14-4973-4b94-97d3-3c0572dcee03\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") "
Sep 17 22:55:04.283194 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.283160    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-tmp-dir\") pod \"19624b14-4973-4b94-97d3-3c0572dcee03\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") "
Sep 17 22:55:04.283194 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.283190    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19624b14-4973-4b94-97d3-3c0572dcee03-tls-certs\") pod \"19624b14-4973-4b94-97d3-3c0572dcee03\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") "
Sep 17 22:55:04.283440 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.283220    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-kserve-provision-location\") pod \"19624b14-4973-4b94-97d3-3c0572dcee03\" (UID: \"19624b14-4973-4b94-97d3-3c0572dcee03\") "
Sep 17 22:55:04.283440 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.283336    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-model-cache" (OuterVolumeSpecName: "model-cache") pod "19624b14-4973-4b94-97d3-3c0572dcee03" (UID: "19624b14-4973-4b94-97d3-3c0572dcee03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:04.283440 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.283405    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19624b14-4973-4b94-97d3-3c0572dcee03-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "19624b14-4973-4b94-97d3-3c0572dcee03" (UID: "19624b14-4973-4b94-97d3-3c0572dcee03"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:55:04.283948 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.283450    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.283948 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.283803    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-home" (OuterVolumeSpecName: "home") pod "19624b14-4973-4b94-97d3-3c0572dcee03" (UID: "19624b14-4973-4b94-97d3-3c0572dcee03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:04.285860 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.285818    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19624b14-4973-4b94-97d3-3c0572dcee03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19624b14-4973-4b94-97d3-3c0572dcee03" (UID: "19624b14-4973-4b94-97d3-3c0572dcee03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:55:04.285964 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.285913    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-dshm" (OuterVolumeSpecName: "dshm") pod "19624b14-4973-4b94-97d3-3c0572dcee03" (UID: "19624b14-4973-4b94-97d3-3c0572dcee03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:04.285964 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.285924    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19624b14-4973-4b94-97d3-3c0572dcee03-kube-api-access-vp6m7" (OuterVolumeSpecName: "kube-api-access-vp6m7") pod "19624b14-4973-4b94-97d3-3c0572dcee03" (UID: "19624b14-4973-4b94-97d3-3c0572dcee03"). InnerVolumeSpecName "kube-api-access-vp6m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:55:04.299932 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.299903    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19624b14-4973-4b94-97d3-3c0572dcee03" (UID: "19624b14-4973-4b94-97d3-3c0572dcee03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:04.306939 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.306911    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19624b14-4973-4b94-97d3-3c0572dcee03" (UID: "19624b14-4973-4b94-97d3-3c0572dcee03"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:04.313128 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.313105    2567 generic.go:356] "Generic (PLEG): container finished" podID="19624b14-4973-4b94-97d3-3c0572dcee03" containerID="be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb" exitCode=0
Sep 17 22:55:04.313128 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.313124    2567 generic.go:356] "Generic (PLEG): container finished" podID="19624b14-4973-4b94-97d3-3c0572dcee03" containerID="fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4" exitCode=0
Sep 17 22:55:04.313328 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.313165    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"
Sep 17 22:55:04.313328 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.313195    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" event={"ID":"19624b14-4973-4b94-97d3-3c0572dcee03","Type":"ContainerDied","Data":"be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb"}
Sep 17 22:55:04.313328 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.313287    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" event={"ID":"19624b14-4973-4b94-97d3-3c0572dcee03","Type":"ContainerDied","Data":"fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4"}
Sep 17 22:55:04.313328 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.313320    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8" event={"ID":"19624b14-4973-4b94-97d3-3c0572dcee03","Type":"ContainerDied","Data":"45ec8451f65ed35cd599a8111f40288c4a670e94831a60cf356f9678888537db"}
Sep 17 22:55:04.313490 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.313354    2567 scope.go:117] "RemoveContainer" containerID="be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb"
Sep 17 22:55:04.322157 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.322139    2567 scope.go:117] "RemoveContainer" containerID="91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615"
Sep 17 22:55:04.329126 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.329111    2567 scope.go:117] "RemoveContainer" containerID="fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4"
Sep 17 22:55:04.335722 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.335699    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"]
Sep 17 22:55:04.336180 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.336163    2567 scope.go:117] "RemoveContainer" containerID="be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb"
Sep 17 22:55:04.336446 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:55:04.336425    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb\": container with ID starting with be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb not found: ID does not exist" containerID="be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb"
Sep 17 22:55:04.336515 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.336458    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb"} err="failed to get container status \"be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb\": rpc error: code = NotFound desc = could not find container \"be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb\": container with ID starting with be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb not found: ID does not exist"
Sep 17 22:55:04.336515 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.336482    2567 scope.go:117] "RemoveContainer" containerID="91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615"
Sep 17 22:55:04.336743 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:55:04.336726    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615\": container with ID starting with 91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615 not found: ID does not exist" containerID="91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615"
Sep 17 22:55:04.336783 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.336749    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615"} err="failed to get container status \"91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615\": rpc error: code = NotFound desc = could not find container \"91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615\": container with ID starting with 91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615 not found: ID does not exist"
Sep 17 22:55:04.336783 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.336764    2567 scope.go:117] "RemoveContainer" containerID="fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4"
Sep 17 22:55:04.336979 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:55:04.336963    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4\": container with ID starting with fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4 not found: ID does not exist" containerID="fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4"
Sep 17 22:55:04.337061 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.336987    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4"} err="failed to get container status \"fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4\": rpc error: code = NotFound desc = could not find container \"fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4\": container with ID starting with fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4 not found: ID does not exist"
Sep 17 22:55:04.337061 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.337017    2567 scope.go:117] "RemoveContainer" containerID="be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb"
Sep 17 22:55:04.337275 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.337259    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb"} err="failed to get container status \"be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb\": rpc error: code = NotFound desc = could not find container \"be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb\": container with ID starting with be030ef5c0de847a58da3384f5c63376b51af1c198d791a9c993aa369fb843bb not found: ID does not exist"
Sep 17 22:55:04.337344 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.337276    2567 scope.go:117] "RemoveContainer" containerID="91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615"
Sep 17 22:55:04.337523 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.337505    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615"} err="failed to get container status \"91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615\": rpc error: code = NotFound desc = could not find container \"91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615\": container with ID starting with 91023a6e9d797425dcc17e3dd30ccf7e6095c5220014770e662318399320a615 not found: ID does not exist"
Sep 17 22:55:04.337572 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.337525    2567 scope.go:117] "RemoveContainer" containerID="fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4"
Sep 17 22:55:04.337741 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.337721    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4"} err="failed to get container status \"fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4\": rpc error: code = NotFound desc = could not find container \"fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4\": container with ID starting with fc1e3527313d35c2b7b24298a1ef7aaa68bfb3e9b055346a2f8a9fa5d1857da4 not found: ID does not exist"
Sep 17 22:55:04.341665 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.341642    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6c8bdcc4c5-9prf8"]
Sep 17 22:55:04.384742 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.384714    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.384742 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.384739    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19624b14-4973-4b94-97d3-3c0572dcee03-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.384742 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.384748    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.385168 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.384756    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19624b14-4973-4b94-97d3-3c0572dcee03-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.385168 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.384765    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.385168 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.384774    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19624b14-4973-4b94-97d3-3c0572dcee03-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:04.385168 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:04.384782    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp6m7\" (UniqueName: \"kubernetes.io/projected/19624b14-4973-4b94-97d3-3c0572dcee03-kube-api-access-vp6m7\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:05.639257 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:05.639225    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" path="/var/lib/kubelet/pods/19624b14-4973-4b94-97d3-3c0572dcee03/volumes"
Sep 17 22:55:43.395141 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:43.395108    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"]
Sep 17 22:55:43.395591 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:43.395286    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg" podUID="ca931302-608d-486f-9984-c0ea421f31e7" containerName="istio-proxy" containerID="cri-o://2181a64f7465116394dafaa712df2e010c558559cef69da0c7215bfd532d2ff0" gracePeriod=30
Sep 17 22:55:43.795197 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:43.795166    2567 scope.go:117] "RemoveContainer" containerID="28337cd77ba52855c2a0745a9b28b0d06138a05f23196733f4956efb416ab5c3"
Sep 17 22:55:43.802772 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:43.802754    2567 scope.go:117] "RemoveContainer" containerID="dbcf97014fe43f9bd7d9cd4b7fe7299a13627db7e4e73594cf639f9138869ac3"
Sep 17 22:55:43.809810 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:43.809785    2567 scope.go:117] "RemoveContainer" containerID="9b68692214cc19ebd88afc36e5b856904a95d5f1bf630618a5f22d0fba6006b0"
Sep 17 22:55:47.632020 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:47.631962    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg" podUID="ca931302-608d-486f-9984-c0ea421f31e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:55:48.642645 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.642622    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:55:48.697252 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.697212    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca931302-608d-486f-9984-c0ea421f31e7-istio-podinfo\") pod \"ca931302-608d-486f-9984-c0ea421f31e7\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") "
Sep 17 22:55:48.697252 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.697276    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-istio-data\") pod \"ca931302-608d-486f-9984-c0ea421f31e7\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") "
Sep 17 22:55:48.697490 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.697292    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca931302-608d-486f-9984-c0ea421f31e7-istio-token\") pod \"ca931302-608d-486f-9984-c0ea421f31e7\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") "
Sep 17 22:55:48.697490 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.697324    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rtsx\" (UniqueName: \"kubernetes.io/projected/ca931302-608d-486f-9984-c0ea421f31e7-kube-api-access-6rtsx\") pod \"ca931302-608d-486f-9984-c0ea421f31e7\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") "
Sep 17 22:55:48.697490 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.697349    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-workload-certs\") pod \"ca931302-608d-486f-9984-c0ea421f31e7\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") "
Sep 17 22:55:48.697490 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.697371    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca931302-608d-486f-9984-c0ea421f31e7-istiod-ca-cert\") pod \"ca931302-608d-486f-9984-c0ea421f31e7\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") "
Sep 17 22:55:48.697490 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.697409    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-istio-envoy\") pod \"ca931302-608d-486f-9984-c0ea421f31e7\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") "
Sep 17 22:55:48.697490 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.697439    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-credential-socket\") pod \"ca931302-608d-486f-9984-c0ea421f31e7\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") "
Sep 17 22:55:48.697490 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.697461    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-workload-socket\") pod \"ca931302-608d-486f-9984-c0ea421f31e7\" (UID: \"ca931302-608d-486f-9984-c0ea421f31e7\") "
Sep 17 22:55:48.697853 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.697569    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca931302-608d-486f-9984-c0ea421f31e7" (UID: "ca931302-608d-486f-9984-c0ea421f31e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:48.697853 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.697695    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-istio-data\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:48.697853 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.697744    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca931302-608d-486f-9984-c0ea421f31e7" (UID: "ca931302-608d-486f-9984-c0ea421f31e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:48.697853 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.697832    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca931302-608d-486f-9984-c0ea421f31e7" (UID: "ca931302-608d-486f-9984-c0ea421f31e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:48.698110 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.698084    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca931302-608d-486f-9984-c0ea421f31e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca931302-608d-486f-9984-c0ea421f31e7" (UID: "ca931302-608d-486f-9984-c0ea421f31e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:55:48.698187 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.698104    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca931302-608d-486f-9984-c0ea421f31e7" (UID: "ca931302-608d-486f-9984-c0ea421f31e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:48.699888 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.699859    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca931302-608d-486f-9984-c0ea421f31e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca931302-608d-486f-9984-c0ea421f31e7" (UID: "ca931302-608d-486f-9984-c0ea421f31e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 22:55:48.700028 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.699967    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca931302-608d-486f-9984-c0ea421f31e7" (UID: "ca931302-608d-486f-9984-c0ea421f31e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:55:48.700335 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.700316    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca931302-608d-486f-9984-c0ea421f31e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca931302-608d-486f-9984-c0ea421f31e7" (UID: "ca931302-608d-486f-9984-c0ea421f31e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:55:48.700402 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.700338    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca931302-608d-486f-9984-c0ea421f31e7-kube-api-access-6rtsx" (OuterVolumeSpecName: "kube-api-access-6rtsx") pod "ca931302-608d-486f-9984-c0ea421f31e7" (UID: "ca931302-608d-486f-9984-c0ea421f31e7"). InnerVolumeSpecName "kube-api-access-6rtsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:55:48.798083 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.798056    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca931302-608d-486f-9984-c0ea421f31e7-istio-podinfo\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:48.798083 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.798080    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca931302-608d-486f-9984-c0ea421f31e7-istio-token\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:48.798260 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.798094    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rtsx\" (UniqueName: \"kubernetes.io/projected/ca931302-608d-486f-9984-c0ea421f31e7-kube-api-access-6rtsx\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:48.798260 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.798106    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-workload-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:48.798260 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.798117    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca931302-608d-486f-9984-c0ea421f31e7-istiod-ca-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:48.798260 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.798126    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-istio-envoy\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:48.798260 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.798137    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-credential-socket\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:48.798260 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:48.798148    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca931302-608d-486f-9984-c0ea421f31e7-workload-socket\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:55:49.474852 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:49.474815    2567 generic.go:356] "Generic (PLEG): container finished" podID="ca931302-608d-486f-9984-c0ea421f31e7" containerID="2181a64f7465116394dafaa712df2e010c558559cef69da0c7215bfd532d2ff0" exitCode=0
Sep 17 22:55:49.474852 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:49.474858    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg" event={"ID":"ca931302-608d-486f-9984-c0ea421f31e7","Type":"ContainerDied","Data":"2181a64f7465116394dafaa712df2e010c558559cef69da0c7215bfd532d2ff0"}
Sep 17 22:55:49.475166 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:49.474871    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"
Sep 17 22:55:49.475166 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:49.474899    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg" event={"ID":"ca931302-608d-486f-9984-c0ea421f31e7","Type":"ContainerDied","Data":"12dbcee93d8312f387ab1c508a226079e89aef78d82beed4817734624b421673"}
Sep 17 22:55:49.475166 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:49.474926    2567 scope.go:117] "RemoveContainer" containerID="2181a64f7465116394dafaa712df2e010c558559cef69da0c7215bfd532d2ff0"
Sep 17 22:55:49.483882 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:49.483864    2567 scope.go:117] "RemoveContainer" containerID="311b772836304d98bc55d631cec03adcc7de67454a93d112c993a6bac4148f18"
Sep 17 22:55:49.491134 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:49.491116    2567 scope.go:117] "RemoveContainer" containerID="2181a64f7465116394dafaa712df2e010c558559cef69da0c7215bfd532d2ff0"
Sep 17 22:55:49.491410 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:55:49.491392    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2181a64f7465116394dafaa712df2e010c558559cef69da0c7215bfd532d2ff0\": container with ID starting with 2181a64f7465116394dafaa712df2e010c558559cef69da0c7215bfd532d2ff0 not found: ID does not exist" containerID="2181a64f7465116394dafaa712df2e010c558559cef69da0c7215bfd532d2ff0"
Sep 17 22:55:49.491478 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:49.491420    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2181a64f7465116394dafaa712df2e010c558559cef69da0c7215bfd532d2ff0"} err="failed to get container status \"2181a64f7465116394dafaa712df2e010c558559cef69da0c7215bfd532d2ff0\": rpc error: code = NotFound desc = could not find container \"2181a64f7465116394dafaa712df2e010c558559cef69da0c7215bfd532d2ff0\": container with ID starting with 2181a64f7465116394dafaa712df2e010c558559cef69da0c7215bfd532d2ff0 not found: ID does not exist"
Sep 17 22:55:49.491478 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:49.491436    2567 scope.go:117] "RemoveContainer" containerID="311b772836304d98bc55d631cec03adcc7de67454a93d112c993a6bac4148f18"
Sep 17 22:55:49.491658 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:55:49.491639    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"311b772836304d98bc55d631cec03adcc7de67454a93d112c993a6bac4148f18\": container with ID starting with 311b772836304d98bc55d631cec03adcc7de67454a93d112c993a6bac4148f18 not found: ID does not exist" containerID="311b772836304d98bc55d631cec03adcc7de67454a93d112c993a6bac4148f18"
Sep 17 22:55:49.491704 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:49.491664    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"311b772836304d98bc55d631cec03adcc7de67454a93d112c993a6bac4148f18"} err="failed to get container status \"311b772836304d98bc55d631cec03adcc7de67454a93d112c993a6bac4148f18\": rpc error: code = NotFound desc = could not find container \"311b772836304d98bc55d631cec03adcc7de67454a93d112c993a6bac4148f18\": container with ID starting with 311b772836304d98bc55d631cec03adcc7de67454a93d112c993a6bac4148f18 not found: ID does not exist"
Sep 17 22:55:49.497352 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:49.497330    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"]
Sep 17 22:55:49.501042 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:49.501011    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-8mtvg"]
Sep 17 22:55:49.638925 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:49.638895    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca931302-608d-486f-9984-c0ea421f31e7" path="/var/lib/kubelet/pods/ca931302-608d-486f-9984-c0ea421f31e7/volumes"
Sep 17 22:55:59.651288 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651258    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"]
Sep 17 22:55:59.651718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651504    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="llm-d-routing-sidecar"
Sep 17 22:55:59.651718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651515    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="llm-d-routing-sidecar"
Sep 17 22:55:59.651718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651527    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="storage-initializer"
Sep 17 22:55:59.651718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651532    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="storage-initializer"
Sep 17 22:55:59.651718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651542    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca931302-608d-486f-9984-c0ea421f31e7" containerName="istio-proxy"
Sep 17 22:55:59.651718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651548    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca931302-608d-486f-9984-c0ea421f31e7" containerName="istio-proxy"
Sep 17 22:55:59.651718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651557    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca931302-608d-486f-9984-c0ea421f31e7" containerName="istio-proxy"
Sep 17 22:55:59.651718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651565    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca931302-608d-486f-9984-c0ea421f31e7" containerName="istio-proxy"
Sep 17 22:55:59.651718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651576    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="main"
Sep 17 22:55:59.651718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651581    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="main"
Sep 17 22:55:59.651718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651626    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca931302-608d-486f-9984-c0ea421f31e7" containerName="istio-proxy"
Sep 17 22:55:59.651718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651633    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="llm-d-routing-sidecar"
Sep 17 22:55:59.651718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651639    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="19624b14-4973-4b94-97d3-3c0572dcee03" containerName="main"
Sep 17 22:55:59.651718 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.651723    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca931302-608d-486f-9984-c0ea421f31e7" containerName="istio-proxy"
Sep 17 22:55:59.654445 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.654428    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.656742 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.656706    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 17 22:55:59.656877 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.656859    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 17 22:55:59.656956 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.656935    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-nmzzx\""
Sep 17 22:55:59.656956 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.656945    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:55:59.657068 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.656962    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 17 22:55:59.657115 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.657084    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 17 22:55:59.665661 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.665641    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"]
Sep 17 22:55:59.668660 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.668638    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad8da491-c82b-4ad7-ae15-78c638e11502-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.668762 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.668670    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8da491-c82b-4ad7-ae15-78c638e11502-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.668762 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.668703    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.668877 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.668760    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.668877 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.668781    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whx8f\" (UniqueName: \"kubernetes.io/projected/ad8da491-c82b-4ad7-ae15-78c638e11502-kube-api-access-whx8f\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.668986 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.668873    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.668986 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.668900    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.668986 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.668936    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.769812 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.769785    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.769916 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.769816    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.769916 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.769833    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whx8f\" (UniqueName: \"kubernetes.io/projected/ad8da491-c82b-4ad7-ae15-78c638e11502-kube-api-access-whx8f\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.769916 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.769872    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.769916 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.769895    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.770147 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.769930    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.770147 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.769979    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad8da491-c82b-4ad7-ae15-78c638e11502-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.770147 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.770021    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8da491-c82b-4ad7-ae15-78c638e11502-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.770304 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.770255    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.770361 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.770311    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.770415 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.770356    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.770467 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.770446    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.770661 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.770644    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad8da491-c82b-4ad7-ae15-78c638e11502-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.772190 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.772172    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.772487 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.772468    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8da491-c82b-4ad7-ae15-78c638e11502-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.777381 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.777361    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whx8f\" (UniqueName: \"kubernetes.io/projected/ad8da491-c82b-4ad7-ae15-78c638e11502-kube-api-access-whx8f\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:55:59.964220 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:55:59.964155    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:56:00.237414 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:00.237386    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"]
Sep 17 22:56:00.239039 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:56:00.239010    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8da491_c82b_4ad7_ae15_78c638e11502.slice/crio-1b237e6e7a39301748f7ea28ebea6ed46a13859717061a3493794b4beb08abae WatchSource:0}: Error finding container 1b237e6e7a39301748f7ea28ebea6ed46a13859717061a3493794b4beb08abae: Status 404 returned error can't find the container with id 1b237e6e7a39301748f7ea28ebea6ed46a13859717061a3493794b4beb08abae
Sep 17 22:56:00.514025 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:00.513920    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" event={"ID":"ad8da491-c82b-4ad7-ae15-78c638e11502","Type":"ContainerStarted","Data":"5550ae4547dcdcbf1875a8bd7ee30378ec9a77b9305e88428708c7bdd2c40ecb"}
Sep 17 22:56:00.514025 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:00.513977    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" event={"ID":"ad8da491-c82b-4ad7-ae15-78c638e11502","Type":"ContainerStarted","Data":"1b237e6e7a39301748f7ea28ebea6ed46a13859717061a3493794b4beb08abae"}
Sep 17 22:56:02.521815 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:02.521782    2567 generic.go:356] "Generic (PLEG): container finished" podID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerID="5550ae4547dcdcbf1875a8bd7ee30378ec9a77b9305e88428708c7bdd2c40ecb" exitCode=0
Sep 17 22:56:02.522270 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:02.521841    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" event={"ID":"ad8da491-c82b-4ad7-ae15-78c638e11502","Type":"ContainerDied","Data":"5550ae4547dcdcbf1875a8bd7ee30378ec9a77b9305e88428708c7bdd2c40ecb"}
Sep 17 22:56:03.527189 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:03.527156    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" event={"ID":"ad8da491-c82b-4ad7-ae15-78c638e11502","Type":"ContainerStarted","Data":"697165a57d53c61e39ef9eb05f7c297034b9d7311af88e2fd45ff1e59d8adbdc"}
Sep 17 22:56:03.547349 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:03.547292    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" podStartSLOduration=2.414514355 podStartE2EDuration="4.547272026s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.132757671s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:56:03.545875956 +0000 UTC m=+3440.554805497" watchObservedRunningTime="2026-09-17 22:56:03.547272026 +0000 UTC m=+3440.556201568"
Sep 17 22:56:03.964475 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:03.964386    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:56:09.965365 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:09.965334    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:56:09.966719 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:09.966686    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 22:56:19.965473 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:19.965433    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 22:56:29.964561 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:29.964517    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 22:56:39.965341 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:39.965299    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 22:56:47.778944 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:47.778908    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"]
Sep 17 22:56:47.783733 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:47.783714    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"
Sep 17 22:56:47.786921 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:47.786882    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-h8zpb\""
Sep 17 22:56:47.787066 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:47.786928    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 22:56:47.787066 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:47.786928    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 22:56:47.787066 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:47.786968    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 22:56:47.787244 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:47.787065    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-w9d94\""
Sep 17 22:56:47.791599 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:47.791578    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"]
Sep 17 22:56:47.838793 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:47.838763    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqr44\" (UniqueName: \"kubernetes.io/projected/a879091f-e741-4314-95b3-976657b39d56-kube-api-access-gqr44\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm\" (UID: \"a879091f-e741-4314-95b3-976657b39d56\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"
Sep 17 22:56:47.838919 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:47.838853    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a879091f-e741-4314-95b3-976657b39d56-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm\" (UID: \"a879091f-e741-4314-95b3-976657b39d56\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"
Sep 17 22:56:47.940192 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:47.940159    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqr44\" (UniqueName: \"kubernetes.io/projected/a879091f-e741-4314-95b3-976657b39d56-kube-api-access-gqr44\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm\" (UID: \"a879091f-e741-4314-95b3-976657b39d56\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"
Sep 17 22:56:47.940303 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:47.940225    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a879091f-e741-4314-95b3-976657b39d56-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm\" (UID: \"a879091f-e741-4314-95b3-976657b39d56\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"
Sep 17 22:56:47.942678 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:47.942659    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a879091f-e741-4314-95b3-976657b39d56-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm\" (UID: \"a879091f-e741-4314-95b3-976657b39d56\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"
Sep 17 22:56:47.948158 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:47.948130    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqr44\" (UniqueName: \"kubernetes.io/projected/a879091f-e741-4314-95b3-976657b39d56-kube-api-access-gqr44\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm\" (UID: \"a879091f-e741-4314-95b3-976657b39d56\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"
Sep 17 22:56:48.102894 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:48.102807    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"
Sep 17 22:56:48.374215 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:48.374113    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"]
Sep 17 22:56:48.692725 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:48.692639    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm" event={"ID":"a879091f-e741-4314-95b3-976657b39d56","Type":"ContainerStarted","Data":"4c71db2efdda0cddd4db58a505c62db72755b89510e4b7772a3ee11532662727"}
Sep 17 22:56:48.692725 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:48.692694    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm" event={"ID":"a879091f-e741-4314-95b3-976657b39d56","Type":"ContainerStarted","Data":"a80c3686c2c97824d51af115a8e25c35647ed8b80c71cc73305b8a226775a058"}
Sep 17 22:56:48.692904 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:48.692749    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"
Sep 17 22:56:48.709931 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:48.709885    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm" podStartSLOduration=1.709870575 podStartE2EDuration="1.709870575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:56:48.708861864 +0000 UTC m=+3485.717791406" watchObservedRunningTime="2026-09-17 22:56:48.709870575 +0000 UTC m=+3485.718800116"
Sep 17 22:56:49.965222 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:49.965178    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 22:56:59.965092 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:56:59.965056    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 22:57:09.965033 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:57:09.964931    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 22:57:19.697714 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:57:19.697681    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"
Sep 17 22:57:19.965566 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:57:19.965479    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 22:57:29.964489 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:57:29.964443    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 22:57:39.965120 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:57:39.965072    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 22:57:49.974180 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:57:49.974150    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:57:49.981679 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:57:49.981653    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:57:54.644342 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:57:54.644307    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"]
Sep 17 22:58:09.666309 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:09.666269    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main" containerID="cri-o://697165a57d53c61e39ef9eb05f7c297034b9d7311af88e2fd45ff1e59d8adbdc" gracePeriod=45
Sep 17 22:58:09.975337 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:09.975236    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 17 22:58:10.409906 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.409881    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:58:10.566206 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.566124    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-home\") pod \"ad8da491-c82b-4ad7-ae15-78c638e11502\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") "
Sep 17 22:58:10.566206 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.566162    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-dshm\") pod \"ad8da491-c82b-4ad7-ae15-78c638e11502\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") "
Sep 17 22:58:10.566206 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.566183    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad8da491-c82b-4ad7-ae15-78c638e11502-cabundle-cert\") pod \"ad8da491-c82b-4ad7-ae15-78c638e11502\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") "
Sep 17 22:58:10.566487 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.566211    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-tmp-dir\") pod \"ad8da491-c82b-4ad7-ae15-78c638e11502\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") "
Sep 17 22:58:10.566487 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.566242    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8da491-c82b-4ad7-ae15-78c638e11502-tls-certs\") pod \"ad8da491-c82b-4ad7-ae15-78c638e11502\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") "
Sep 17 22:58:10.566487 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.566275    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-kserve-provision-location\") pod \"ad8da491-c82b-4ad7-ae15-78c638e11502\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") "
Sep 17 22:58:10.566487 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.566310    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whx8f\" (UniqueName: \"kubernetes.io/projected/ad8da491-c82b-4ad7-ae15-78c638e11502-kube-api-access-whx8f\") pod \"ad8da491-c82b-4ad7-ae15-78c638e11502\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") "
Sep 17 22:58:10.566487 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.566446    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-model-cache\") pod \"ad8da491-c82b-4ad7-ae15-78c638e11502\" (UID: \"ad8da491-c82b-4ad7-ae15-78c638e11502\") "
Sep 17 22:58:10.566928 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.566880    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-home" (OuterVolumeSpecName: "home") pod "ad8da491-c82b-4ad7-ae15-78c638e11502" (UID: "ad8da491-c82b-4ad7-ae15-78c638e11502"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:58:10.567037 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.566982    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8da491-c82b-4ad7-ae15-78c638e11502-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad8da491-c82b-4ad7-ae15-78c638e11502" (UID: "ad8da491-c82b-4ad7-ae15-78c638e11502"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:58:10.567090 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.567073    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad8da491-c82b-4ad7-ae15-78c638e11502" (UID: "ad8da491-c82b-4ad7-ae15-78c638e11502"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:58:10.568588 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.568561    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8da491-c82b-4ad7-ae15-78c638e11502-kube-api-access-whx8f" (OuterVolumeSpecName: "kube-api-access-whx8f") pod "ad8da491-c82b-4ad7-ae15-78c638e11502" (UID: "ad8da491-c82b-4ad7-ae15-78c638e11502"). InnerVolumeSpecName "kube-api-access-whx8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:58:10.568680 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.568671    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-dshm" (OuterVolumeSpecName: "dshm") pod "ad8da491-c82b-4ad7-ae15-78c638e11502" (UID: "ad8da491-c82b-4ad7-ae15-78c638e11502"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:58:10.568859 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.568844    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8da491-c82b-4ad7-ae15-78c638e11502-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad8da491-c82b-4ad7-ae15-78c638e11502" (UID: "ad8da491-c82b-4ad7-ae15-78c638e11502"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:58:10.584948 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.584919    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad8da491-c82b-4ad7-ae15-78c638e11502" (UID: "ad8da491-c82b-4ad7-ae15-78c638e11502"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:58:10.595765 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.595744    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad8da491-c82b-4ad7-ae15-78c638e11502" (UID: "ad8da491-c82b-4ad7-ae15-78c638e11502"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:58:10.667836 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.667814    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:58:10.667836 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.667833    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:58:10.668180 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.667842    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad8da491-c82b-4ad7-ae15-78c638e11502-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:58:10.668180 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.667851    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:58:10.668180 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.667859    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8da491-c82b-4ad7-ae15-78c638e11502-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:58:10.668180 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.667866    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:58:10.668180 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.667874    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whx8f\" (UniqueName: \"kubernetes.io/projected/ad8da491-c82b-4ad7-ae15-78c638e11502-kube-api-access-whx8f\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:58:10.668180 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.667883    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8da491-c82b-4ad7-ae15-78c638e11502-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:58:10.999806 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.999773    2567 generic.go:356] "Generic (PLEG): container finished" podID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerID="697165a57d53c61e39ef9eb05f7c297034b9d7311af88e2fd45ff1e59d8adbdc" exitCode=0
Sep 17 22:58:10.999972 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.999832    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"
Sep 17 22:58:10.999972 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.999856    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" event={"ID":"ad8da491-c82b-4ad7-ae15-78c638e11502","Type":"ContainerDied","Data":"697165a57d53c61e39ef9eb05f7c297034b9d7311af88e2fd45ff1e59d8adbdc"}
Sep 17 22:58:10.999972 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.999908    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x" event={"ID":"ad8da491-c82b-4ad7-ae15-78c638e11502","Type":"ContainerDied","Data":"1b237e6e7a39301748f7ea28ebea6ed46a13859717061a3493794b4beb08abae"}
Sep 17 22:58:10.999972 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:10.999929    2567 scope.go:117] "RemoveContainer" containerID="697165a57d53c61e39ef9eb05f7c297034b9d7311af88e2fd45ff1e59d8adbdc"
Sep 17 22:58:11.008056 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:11.007870    2567 scope.go:117] "RemoveContainer" containerID="5550ae4547dcdcbf1875a8bd7ee30378ec9a77b9305e88428708c7bdd2c40ecb"
Sep 17 22:58:11.016234 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:11.016201    2567 scope.go:117] "RemoveContainer" containerID="697165a57d53c61e39ef9eb05f7c297034b9d7311af88e2fd45ff1e59d8adbdc"
Sep 17 22:58:11.016688 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:58:11.016665    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697165a57d53c61e39ef9eb05f7c297034b9d7311af88e2fd45ff1e59d8adbdc\": container with ID starting with 697165a57d53c61e39ef9eb05f7c297034b9d7311af88e2fd45ff1e59d8adbdc not found: ID does not exist" containerID="697165a57d53c61e39ef9eb05f7c297034b9d7311af88e2fd45ff1e59d8adbdc"
Sep 17 22:58:11.016755 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:11.016698    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697165a57d53c61e39ef9eb05f7c297034b9d7311af88e2fd45ff1e59d8adbdc"} err="failed to get container status \"697165a57d53c61e39ef9eb05f7c297034b9d7311af88e2fd45ff1e59d8adbdc\": rpc error: code = NotFound desc = could not find container \"697165a57d53c61e39ef9eb05f7c297034b9d7311af88e2fd45ff1e59d8adbdc\": container with ID starting with 697165a57d53c61e39ef9eb05f7c297034b9d7311af88e2fd45ff1e59d8adbdc not found: ID does not exist"
Sep 17 22:58:11.016755 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:11.016721    2567 scope.go:117] "RemoveContainer" containerID="5550ae4547dcdcbf1875a8bd7ee30378ec9a77b9305e88428708c7bdd2c40ecb"
Sep 17 22:58:11.017013 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:58:11.016978    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5550ae4547dcdcbf1875a8bd7ee30378ec9a77b9305e88428708c7bdd2c40ecb\": container with ID starting with 5550ae4547dcdcbf1875a8bd7ee30378ec9a77b9305e88428708c7bdd2c40ecb not found: ID does not exist" containerID="5550ae4547dcdcbf1875a8bd7ee30378ec9a77b9305e88428708c7bdd2c40ecb"
Sep 17 22:58:11.017062 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:11.017017    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5550ae4547dcdcbf1875a8bd7ee30378ec9a77b9305e88428708c7bdd2c40ecb"} err="failed to get container status \"5550ae4547dcdcbf1875a8bd7ee30378ec9a77b9305e88428708c7bdd2c40ecb\": rpc error: code = NotFound desc = could not find container \"5550ae4547dcdcbf1875a8bd7ee30378ec9a77b9305e88428708c7bdd2c40ecb\": container with ID starting with 5550ae4547dcdcbf1875a8bd7ee30378ec9a77b9305e88428708c7bdd2c40ecb not found: ID does not exist"
Sep 17 22:58:11.020680 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:11.020657    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"]
Sep 17 22:58:11.025106 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:11.025087    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7596879964wpj6x"]
Sep 17 22:58:11.639136 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:11.639099    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" path="/var/lib/kubelet/pods/ad8da491-c82b-4ad7-ae15-78c638e11502/volumes"
Sep 17 22:58:17.441347 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.441315    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"]
Sep 17 22:58:17.441796 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.441584    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="storage-initializer"
Sep 17 22:58:17.441796 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.441596    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="storage-initializer"
Sep 17 22:58:17.441796 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.441612    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main"
Sep 17 22:58:17.441796 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.441618    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main"
Sep 17 22:58:17.441796 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.441683    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8da491-c82b-4ad7-ae15-78c638e11502" containerName="main"
Sep 17 22:58:17.445847 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.445804    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"
Sep 17 22:58:17.448425 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.448401    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 22:58:17.449485 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.449464    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 22:58:17.449485 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.449479    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 22:58:17.449646 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.449577    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-mjx66\""
Sep 17 22:58:17.449646 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.449479    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-zmcpl\""
Sep 17 22:58:17.456902 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.456882    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"]
Sep 17 22:58:17.502327 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.502302    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 22:58:17.505458 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.505441    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.507766 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.507747    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 22:58:17.508059 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.508042    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-74x5g\""
Sep 17 22:58:17.508246 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.508230    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:58:17.517939 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.517919    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 22:58:17.614283 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.614256    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.614283 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.614283    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.614478 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.614314    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.614478 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.614334    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f9a32ca-b16c-4279-824a-541c7090a74d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.614478 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.614396    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.614478 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.614469    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thl5n\" (UniqueName: \"kubernetes.io/projected/717ddc46-053e-4ec8-a325-4f4cf52a1121-kube-api-access-thl5n\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x\" (UID: \"717ddc46-053e-4ec8-a325-4f4cf52a1121\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"
Sep 17 22:58:17.614666 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.614497    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.614666 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.614518    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9a32ca-b16c-4279-824a-541c7090a74d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.614666 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.614534    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/717ddc46-053e-4ec8-a325-4f4cf52a1121-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x\" (UID: \"717ddc46-053e-4ec8-a325-4f4cf52a1121\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"
Sep 17 22:58:17.614666 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.614597    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6x9\" (UniqueName: \"kubernetes.io/projected/8f9a32ca-b16c-4279-824a-541c7090a74d-kube-api-access-gt6x9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.715356 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.715297    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/717ddc46-053e-4ec8-a325-4f4cf52a1121-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x\" (UID: \"717ddc46-053e-4ec8-a325-4f4cf52a1121\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"
Sep 17 22:58:17.715356 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.715329    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6x9\" (UniqueName: \"kubernetes.io/projected/8f9a32ca-b16c-4279-824a-541c7090a74d-kube-api-access-gt6x9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.715519 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.715364    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.715519 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.715389    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.715519 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.715434    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.715519 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.715456    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f9a32ca-b16c-4279-824a-541c7090a74d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.715519 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.715487    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.715774 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.715610    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thl5n\" (UniqueName: \"kubernetes.io/projected/717ddc46-053e-4ec8-a325-4f4cf52a1121-kube-api-access-thl5n\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x\" (UID: \"717ddc46-053e-4ec8-a325-4f4cf52a1121\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"
Sep 17 22:58:17.715774 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.715649    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.715774 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.715677    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9a32ca-b16c-4279-824a-541c7090a74d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.715922 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.715829    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.715922 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.715901    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.716057 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.716044    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.716113 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.716080    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.716177 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.716121    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f9a32ca-b16c-4279-824a-541c7090a74d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.717589 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.717558    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.717929 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.717913    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/717ddc46-053e-4ec8-a325-4f4cf52a1121-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x\" (UID: \"717ddc46-053e-4ec8-a325-4f4cf52a1121\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"
Sep 17 22:58:17.717972 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.717961    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9a32ca-b16c-4279-824a-541c7090a74d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.722991 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.722967    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6x9\" (UniqueName: \"kubernetes.io/projected/8f9a32ca-b16c-4279-824a-541c7090a74d-kube-api-access-gt6x9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:17.723811 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.723791    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thl5n\" (UniqueName: \"kubernetes.io/projected/717ddc46-053e-4ec8-a325-4f4cf52a1121-kube-api-access-thl5n\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x\" (UID: \"717ddc46-053e-4ec8-a325-4f4cf52a1121\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"
Sep 17 22:58:17.755396 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.755373    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"
Sep 17 22:58:17.814793 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:17.814763    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 22:58:18.032596 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:18.032566    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"]
Sep 17 22:58:18.034906 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:58:18.034869    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717ddc46_053e_4ec8_a325_4f4cf52a1121.slice/crio-e4bbf3daa33e522f934d32671c7058e30c557a8f20a04cd2d62cfb77a8c84cfa WatchSource:0}: Error finding container e4bbf3daa33e522f934d32671c7058e30c557a8f20a04cd2d62cfb77a8c84cfa: Status 404 returned error can't find the container with id e4bbf3daa33e522f934d32671c7058e30c557a8f20a04cd2d62cfb77a8c84cfa
Sep 17 22:58:18.103018 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:18.102967    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 22:58:18.105562 ip-10-0-142-197 kubenswrapper[2567]: W0917 22:58:18.105531    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9a32ca_b16c_4279_824a_541c7090a74d.slice/crio-1c254adf3c80aac19d3fa28a45a3fb09304e7285130ae58fb8ce03147daacbf4 WatchSource:0}: Error finding container 1c254adf3c80aac19d3fa28a45a3fb09304e7285130ae58fb8ce03147daacbf4: Status 404 returned error can't find the container with id 1c254adf3c80aac19d3fa28a45a3fb09304e7285130ae58fb8ce03147daacbf4
Sep 17 22:58:19.030467 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:19.030431    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x" event={"ID":"717ddc46-053e-4ec8-a325-4f4cf52a1121","Type":"ContainerStarted","Data":"7a18af3998af0e6f3fcec4fc11f4e5a758f3e731ef0a0192a72f178c5f3b27f2"}
Sep 17 22:58:19.030941 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:19.030485    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x" event={"ID":"717ddc46-053e-4ec8-a325-4f4cf52a1121","Type":"ContainerStarted","Data":"e4bbf3daa33e522f934d32671c7058e30c557a8f20a04cd2d62cfb77a8c84cfa"}
Sep 17 22:58:19.030941 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:19.030592    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"
Sep 17 22:58:19.031866 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:19.031839    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"8f9a32ca-b16c-4279-824a-541c7090a74d","Type":"ContainerStarted","Data":"0ab7d1fb47feff802281b90d550fbe10a5f691a9de73cdf7aa85fe8d18e8ed89"}
Sep 17 22:58:19.031981 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:19.031887    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"8f9a32ca-b16c-4279-824a-541c7090a74d","Type":"ContainerStarted","Data":"1c254adf3c80aac19d3fa28a45a3fb09304e7285130ae58fb8ce03147daacbf4"}
Sep 17 22:58:19.046133 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:19.046073    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x" podStartSLOduration=2.0460574559999998 podStartE2EDuration="2.046057456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:58:19.045179951 +0000 UTC m=+3576.054109492" watchObservedRunningTime="2026-09-17 22:58:19.046057456 +0000 UTC m=+3576.054986997"
Sep 17 22:58:21.041018 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:21.040963    2567 generic.go:356] "Generic (PLEG): container finished" podID="8f9a32ca-b16c-4279-824a-541c7090a74d" containerID="0ab7d1fb47feff802281b90d550fbe10a5f691a9de73cdf7aa85fe8d18e8ed89" exitCode=0
Sep 17 22:58:21.041387 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:21.041032    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"8f9a32ca-b16c-4279-824a-541c7090a74d","Type":"ContainerDied","Data":"0ab7d1fb47feff802281b90d550fbe10a5f691a9de73cdf7aa85fe8d18e8ed89"}
Sep 17 22:58:22.047280 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:22.047246    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"8f9a32ca-b16c-4279-824a-541c7090a74d","Type":"ContainerStarted","Data":"6b47fbe5a00b3f17f87b388a21854417ec53f092e3ba31081e5baa271f8e7b02"}
Sep 17 22:58:22.068515 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:22.068459    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.999209435 podStartE2EDuration="5.068444545s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.06923511s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:58:22.066186609 +0000 UTC m=+3579.075116184" watchObservedRunningTime="2026-09-17 22:58:22.068444545 +0000 UTC m=+3579.077374085"
Sep 17 22:58:50.037500 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:50.037464    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"
Sep 17 22:58:52.672842 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:52.672807    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"]
Sep 17 22:58:59.698196 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:58:59.698158    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm" podUID="a879091f-e741-4314-95b3-976657b39d56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:59:07.691344 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:07.691297    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm" podUID="a879091f-e741-4314-95b3-976657b39d56" containerName="main" containerID="cri-o://4c71db2efdda0cddd4db58a505c62db72755b89510e4b7772a3ee11532662727" gracePeriod=45
Sep 17 22:59:09.697779 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:09.697738    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm" podUID="a879091f-e741-4314-95b3-976657b39d56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:59:19.697903 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:19.697867    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm" podUID="a879091f-e741-4314-95b3-976657b39d56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:59:19.698393 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:19.697987    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"
Sep 17 22:59:29.697165 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:29.697119    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm" podUID="a879091f-e741-4314-95b3-976657b39d56" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 22:59:37.943986 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:37.943960    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"
Sep 17 22:59:38.034099 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.034061    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a879091f-e741-4314-95b3-976657b39d56-tls-certs\") pod \"a879091f-e741-4314-95b3-976657b39d56\" (UID: \"a879091f-e741-4314-95b3-976657b39d56\") "
Sep 17 22:59:38.034294 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.034117    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqr44\" (UniqueName: \"kubernetes.io/projected/a879091f-e741-4314-95b3-976657b39d56-kube-api-access-gqr44\") pod \"a879091f-e741-4314-95b3-976657b39d56\" (UID: \"a879091f-e741-4314-95b3-976657b39d56\") "
Sep 17 22:59:38.036319 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.036286    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a879091f-e741-4314-95b3-976657b39d56-kube-api-access-gqr44" (OuterVolumeSpecName: "kube-api-access-gqr44") pod "a879091f-e741-4314-95b3-976657b39d56" (UID: "a879091f-e741-4314-95b3-976657b39d56"). InnerVolumeSpecName "kube-api-access-gqr44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:59:38.036456 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.036317    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a879091f-e741-4314-95b3-976657b39d56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a879091f-e741-4314-95b3-976657b39d56" (UID: "a879091f-e741-4314-95b3-976657b39d56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:59:38.135139 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.135111    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a879091f-e741-4314-95b3-976657b39d56-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:59:38.135139 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.135135    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqr44\" (UniqueName: \"kubernetes.io/projected/a879091f-e741-4314-95b3-976657b39d56-kube-api-access-gqr44\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 22:59:38.346164 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.346063    2567 generic.go:356] "Generic (PLEG): container finished" podID="a879091f-e741-4314-95b3-976657b39d56" containerID="4c71db2efdda0cddd4db58a505c62db72755b89510e4b7772a3ee11532662727" exitCode=0
Sep 17 22:59:38.346164 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.346105    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"
Sep 17 22:59:38.346164 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.346141    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm" event={"ID":"a879091f-e741-4314-95b3-976657b39d56","Type":"ContainerDied","Data":"4c71db2efdda0cddd4db58a505c62db72755b89510e4b7772a3ee11532662727"}
Sep 17 22:59:38.346407 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.346188    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm" event={"ID":"a879091f-e741-4314-95b3-976657b39d56","Type":"ContainerDied","Data":"a80c3686c2c97824d51af115a8e25c35647ed8b80c71cc73305b8a226775a058"}
Sep 17 22:59:38.346407 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.346208    2567 scope.go:117] "RemoveContainer" containerID="4c71db2efdda0cddd4db58a505c62db72755b89510e4b7772a3ee11532662727"
Sep 17 22:59:38.357675 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.357655    2567 scope.go:117] "RemoveContainer" containerID="4c71db2efdda0cddd4db58a505c62db72755b89510e4b7772a3ee11532662727"
Sep 17 22:59:38.357935 ip-10-0-142-197 kubenswrapper[2567]: E0917 22:59:38.357915    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c71db2efdda0cddd4db58a505c62db72755b89510e4b7772a3ee11532662727\": container with ID starting with 4c71db2efdda0cddd4db58a505c62db72755b89510e4b7772a3ee11532662727 not found: ID does not exist" containerID="4c71db2efdda0cddd4db58a505c62db72755b89510e4b7772a3ee11532662727"
Sep 17 22:59:38.357982 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.357942    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c71db2efdda0cddd4db58a505c62db72755b89510e4b7772a3ee11532662727"} err="failed to get container status \"4c71db2efdda0cddd4db58a505c62db72755b89510e4b7772a3ee11532662727\": rpc error: code = NotFound desc = could not find container \"4c71db2efdda0cddd4db58a505c62db72755b89510e4b7772a3ee11532662727\": container with ID starting with 4c71db2efdda0cddd4db58a505c62db72755b89510e4b7772a3ee11532662727 not found: ID does not exist"
Sep 17 22:59:38.367904 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.367881    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"]
Sep 17 22:59:38.371942 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:38.371923    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-gg9fm"]
Sep 17 22:59:39.640390 ip-10-0-142-197 kubenswrapper[2567]: I0917 22:59:39.640358    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a879091f-e741-4314-95b3-976657b39d56" path="/var/lib/kubelet/pods/a879091f-e741-4314-95b3-976657b39d56/volumes"
Sep 17 23:00:16.170321 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:16.170236    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"]
Sep 17 23:00:20.037738 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:20.037706    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x" podUID="717ddc46-053e-4ec8-a325-4f4cf52a1121" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:00:30.037727 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:30.037691    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x" podUID="717ddc46-053e-4ec8-a325-4f4cf52a1121" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:00:31.188749 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:31.188709    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x" podUID="717ddc46-053e-4ec8-a325-4f4cf52a1121" containerName="main" containerID="cri-o://7a18af3998af0e6f3fcec4fc11f4e5a758f3e731ef0a0192a72f178c5f3b27f2" gracePeriod=45
Sep 17 23:00:32.796544 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:32.796499    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 23:00:40.038051 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:40.037992    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x" podUID="717ddc46-053e-4ec8-a325-4f4cf52a1121" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:00:40.038423 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:40.038127    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"
Sep 17 23:00:47.818273 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:47.818225    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="8f9a32ca-b16c-4279-824a-541c7090a74d" containerName="main" containerID="cri-o://6b47fbe5a00b3f17f87b388a21854417ec53f092e3ba31081e5baa271f8e7b02" gracePeriod=45
Sep 17 23:00:48.473508 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.473485    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 23:00:48.523087 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.523062    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt6x9\" (UniqueName: \"kubernetes.io/projected/8f9a32ca-b16c-4279-824a-541c7090a74d-kube-api-access-gt6x9\") pod \"8f9a32ca-b16c-4279-824a-541c7090a74d\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") "
Sep 17 23:00:48.523239 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.523104    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-kserve-provision-location\") pod \"8f9a32ca-b16c-4279-824a-541c7090a74d\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") "
Sep 17 23:00:48.523239 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.523177    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-dshm\") pod \"8f9a32ca-b16c-4279-824a-541c7090a74d\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") "
Sep 17 23:00:48.523239 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.523202    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-model-cache\") pod \"8f9a32ca-b16c-4279-824a-541c7090a74d\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") "
Sep 17 23:00:48.523239 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.523233    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-tmp-dir\") pod \"8f9a32ca-b16c-4279-824a-541c7090a74d\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") "
Sep 17 23:00:48.523448 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.523267    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-home\") pod \"8f9a32ca-b16c-4279-824a-541c7090a74d\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") "
Sep 17 23:00:48.523448 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.523304    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f9a32ca-b16c-4279-824a-541c7090a74d-cabundle-cert\") pod \"8f9a32ca-b16c-4279-824a-541c7090a74d\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") "
Sep 17 23:00:48.523448 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.523328    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9a32ca-b16c-4279-824a-541c7090a74d-tls-certs\") pod \"8f9a32ca-b16c-4279-824a-541c7090a74d\" (UID: \"8f9a32ca-b16c-4279-824a-541c7090a74d\") "
Sep 17 23:00:48.523585 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.523436    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f9a32ca-b16c-4279-824a-541c7090a74d" (UID: "8f9a32ca-b16c-4279-824a-541c7090a74d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:00:48.523585 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.523540    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:00:48.523814 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.523788    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9a32ca-b16c-4279-824a-541c7090a74d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f9a32ca-b16c-4279-824a-541c7090a74d" (UID: "8f9a32ca-b16c-4279-824a-541c7090a74d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:00:48.524069 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.524042    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-home" (OuterVolumeSpecName: "home") pod "8f9a32ca-b16c-4279-824a-541c7090a74d" (UID: "8f9a32ca-b16c-4279-824a-541c7090a74d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:00:48.525468 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.525446    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f9a32ca-b16c-4279-824a-541c7090a74d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f9a32ca-b16c-4279-824a-541c7090a74d" (UID: "8f9a32ca-b16c-4279-824a-541c7090a74d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:00:48.525468 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.525453    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-dshm" (OuterVolumeSpecName: "dshm") pod "8f9a32ca-b16c-4279-824a-541c7090a74d" (UID: "8f9a32ca-b16c-4279-824a-541c7090a74d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:00:48.525681 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.525665    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9a32ca-b16c-4279-824a-541c7090a74d-kube-api-access-gt6x9" (OuterVolumeSpecName: "kube-api-access-gt6x9") pod "8f9a32ca-b16c-4279-824a-541c7090a74d" (UID: "8f9a32ca-b16c-4279-824a-541c7090a74d"). InnerVolumeSpecName "kube-api-access-gt6x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:00:48.535439 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.535415    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f9a32ca-b16c-4279-824a-541c7090a74d" (UID: "8f9a32ca-b16c-4279-824a-541c7090a74d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:00:48.546605 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.546585    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f9a32ca-b16c-4279-824a-541c7090a74d" (UID: "8f9a32ca-b16c-4279-824a-541c7090a74d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:00:48.609633 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.609570    2567 generic.go:356] "Generic (PLEG): container finished" podID="8f9a32ca-b16c-4279-824a-541c7090a74d" containerID="6b47fbe5a00b3f17f87b388a21854417ec53f092e3ba31081e5baa271f8e7b02" exitCode=0
Sep 17 23:00:48.609633 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.609616    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"8f9a32ca-b16c-4279-824a-541c7090a74d","Type":"ContainerDied","Data":"6b47fbe5a00b3f17f87b388a21854417ec53f092e3ba31081e5baa271f8e7b02"}
Sep 17 23:00:48.609633 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.609626    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 23:00:48.609832 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.609655    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"8f9a32ca-b16c-4279-824a-541c7090a74d","Type":"ContainerDied","Data":"1c254adf3c80aac19d3fa28a45a3fb09304e7285130ae58fb8ce03147daacbf4"}
Sep 17 23:00:48.609832 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.609676    2567 scope.go:117] "RemoveContainer" containerID="6b47fbe5a00b3f17f87b388a21854417ec53f092e3ba31081e5baa271f8e7b02"
Sep 17 23:00:48.617704 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.617688    2567 scope.go:117] "RemoveContainer" containerID="0ab7d1fb47feff802281b90d550fbe10a5f691a9de73cdf7aa85fe8d18e8ed89"
Sep 17 23:00:48.624263 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.624246    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:00:48.624341 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.624264    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:00:48.624341 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.624273    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f9a32ca-b16c-4279-824a-541c7090a74d-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:00:48.624341 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.624280    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f9a32ca-b16c-4279-824a-541c7090a74d-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:00:48.624341 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.624288    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt6x9\" (UniqueName: \"kubernetes.io/projected/8f9a32ca-b16c-4279-824a-541c7090a74d-kube-api-access-gt6x9\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:00:48.624341 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.624295    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:00:48.624341 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.624303    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f9a32ca-b16c-4279-824a-541c7090a74d-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:00:48.624627 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.624614    2567 scope.go:117] "RemoveContainer" containerID="6b47fbe5a00b3f17f87b388a21854417ec53f092e3ba31081e5baa271f8e7b02"
Sep 17 23:00:48.624866 ip-10-0-142-197 kubenswrapper[2567]: E0917 23:00:48.624850    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b47fbe5a00b3f17f87b388a21854417ec53f092e3ba31081e5baa271f8e7b02\": container with ID starting with 6b47fbe5a00b3f17f87b388a21854417ec53f092e3ba31081e5baa271f8e7b02 not found: ID does not exist" containerID="6b47fbe5a00b3f17f87b388a21854417ec53f092e3ba31081e5baa271f8e7b02"
Sep 17 23:00:48.624906 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.624872    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b47fbe5a00b3f17f87b388a21854417ec53f092e3ba31081e5baa271f8e7b02"} err="failed to get container status \"6b47fbe5a00b3f17f87b388a21854417ec53f092e3ba31081e5baa271f8e7b02\": rpc error: code = NotFound desc = could not find container \"6b47fbe5a00b3f17f87b388a21854417ec53f092e3ba31081e5baa271f8e7b02\": container with ID starting with 6b47fbe5a00b3f17f87b388a21854417ec53f092e3ba31081e5baa271f8e7b02 not found: ID does not exist"
Sep 17 23:00:48.624906 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.624885    2567 scope.go:117] "RemoveContainer" containerID="0ab7d1fb47feff802281b90d550fbe10a5f691a9de73cdf7aa85fe8d18e8ed89"
Sep 17 23:00:48.625140 ip-10-0-142-197 kubenswrapper[2567]: E0917 23:00:48.625123    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab7d1fb47feff802281b90d550fbe10a5f691a9de73cdf7aa85fe8d18e8ed89\": container with ID starting with 0ab7d1fb47feff802281b90d550fbe10a5f691a9de73cdf7aa85fe8d18e8ed89 not found: ID does not exist" containerID="0ab7d1fb47feff802281b90d550fbe10a5f691a9de73cdf7aa85fe8d18e8ed89"
Sep 17 23:00:48.625184 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.625144    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab7d1fb47feff802281b90d550fbe10a5f691a9de73cdf7aa85fe8d18e8ed89"} err="failed to get container status \"0ab7d1fb47feff802281b90d550fbe10a5f691a9de73cdf7aa85fe8d18e8ed89\": rpc error: code = NotFound desc = could not find container \"0ab7d1fb47feff802281b90d550fbe10a5f691a9de73cdf7aa85fe8d18e8ed89\": container with ID starting with 0ab7d1fb47feff802281b90d550fbe10a5f691a9de73cdf7aa85fe8d18e8ed89 not found: ID does not exist"
Sep 17 23:00:48.630801 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.630782    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 23:00:48.635925 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:48.635906    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 23:00:49.642209 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:49.640271    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f9a32ca-b16c-4279-824a-541c7090a74d" path="/var/lib/kubelet/pods/8f9a32ca-b16c-4279-824a-541c7090a74d/volumes"
Sep 17 23:00:50.037145 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:00:50.037112    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x" podUID="717ddc46-053e-4ec8-a325-4f4cf52a1121" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:01:00.037569 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:00.037531    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x" podUID="717ddc46-053e-4ec8-a325-4f4cf52a1121" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:01:01.655020 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:01.654966    2567 generic.go:356] "Generic (PLEG): container finished" podID="717ddc46-053e-4ec8-a325-4f4cf52a1121" containerID="7a18af3998af0e6f3fcec4fc11f4e5a758f3e731ef0a0192a72f178c5f3b27f2" exitCode=0
Sep 17 23:01:01.655398 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:01.655038    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x" event={"ID":"717ddc46-053e-4ec8-a325-4f4cf52a1121","Type":"ContainerDied","Data":"7a18af3998af0e6f3fcec4fc11f4e5a758f3e731ef0a0192a72f178c5f3b27f2"}
Sep 17 23:01:01.830153 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:01.830127    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"
Sep 17 23:01:01.911134 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:01.911037    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/717ddc46-053e-4ec8-a325-4f4cf52a1121-tls-certs\") pod \"717ddc46-053e-4ec8-a325-4f4cf52a1121\" (UID: \"717ddc46-053e-4ec8-a325-4f4cf52a1121\") "
Sep 17 23:01:01.911134 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:01.911112    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thl5n\" (UniqueName: \"kubernetes.io/projected/717ddc46-053e-4ec8-a325-4f4cf52a1121-kube-api-access-thl5n\") pod \"717ddc46-053e-4ec8-a325-4f4cf52a1121\" (UID: \"717ddc46-053e-4ec8-a325-4f4cf52a1121\") "
Sep 17 23:01:01.913188 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:01.913161    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717ddc46-053e-4ec8-a325-4f4cf52a1121-kube-api-access-thl5n" (OuterVolumeSpecName: "kube-api-access-thl5n") pod "717ddc46-053e-4ec8-a325-4f4cf52a1121" (UID: "717ddc46-053e-4ec8-a325-4f4cf52a1121"). InnerVolumeSpecName "kube-api-access-thl5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:01:01.913293 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:01.913168    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/717ddc46-053e-4ec8-a325-4f4cf52a1121-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "717ddc46-053e-4ec8-a325-4f4cf52a1121" (UID: "717ddc46-053e-4ec8-a325-4f4cf52a1121"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:01:02.012467 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:02.012421    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thl5n\" (UniqueName: \"kubernetes.io/projected/717ddc46-053e-4ec8-a325-4f4cf52a1121-kube-api-access-thl5n\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:01:02.012467 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:02.012459    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/717ddc46-053e-4ec8-a325-4f4cf52a1121-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:01:02.660130 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:02.660092    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x" event={"ID":"717ddc46-053e-4ec8-a325-4f4cf52a1121","Type":"ContainerDied","Data":"e4bbf3daa33e522f934d32671c7058e30c557a8f20a04cd2d62cfb77a8c84cfa"}
Sep 17 23:01:02.660130 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:02.660125    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"
Sep 17 23:01:02.660676 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:02.660156    2567 scope.go:117] "RemoveContainer" containerID="7a18af3998af0e6f3fcec4fc11f4e5a758f3e731ef0a0192a72f178c5f3b27f2"
Sep 17 23:01:02.680915 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:02.680888    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"]
Sep 17 23:01:02.684532 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:02.684511    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schemhf2x"]
Sep 17 23:01:03.643430 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:03.643396    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="717ddc46-053e-4ec8-a325-4f4cf52a1121" path="/var/lib/kubelet/pods/717ddc46-053e-4ec8-a325-4f4cf52a1121/volumes"
Sep 17 23:01:18.741487 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.741450    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"]
Sep 17 23:01:18.741980 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.741790    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9a32ca-b16c-4279-824a-541c7090a74d" containerName="storage-initializer"
Sep 17 23:01:18.741980 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.741809    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9a32ca-b16c-4279-824a-541c7090a74d" containerName="storage-initializer"
Sep 17 23:01:18.741980 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.741826    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="717ddc46-053e-4ec8-a325-4f4cf52a1121" containerName="main"
Sep 17 23:01:18.741980 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.741834    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="717ddc46-053e-4ec8-a325-4f4cf52a1121" containerName="main"
Sep 17 23:01:18.741980 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.741844    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a879091f-e741-4314-95b3-976657b39d56" containerName="main"
Sep 17 23:01:18.741980 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.741854    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a879091f-e741-4314-95b3-976657b39d56" containerName="main"
Sep 17 23:01:18.741980 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.741875    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9a32ca-b16c-4279-824a-541c7090a74d" containerName="main"
Sep 17 23:01:18.741980 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.741882    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9a32ca-b16c-4279-824a-541c7090a74d" containerName="main"
Sep 17 23:01:18.741980 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.741944    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a879091f-e741-4314-95b3-976657b39d56" containerName="main"
Sep 17 23:01:18.741980 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.741952    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9a32ca-b16c-4279-824a-541c7090a74d" containerName="main"
Sep 17 23:01:18.741980 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.741962    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="717ddc46-053e-4ec8-a325-4f4cf52a1121" containerName="main"
Sep 17 23:01:18.746142 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.746122    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.749586 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.749560    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-2h7dh\""
Sep 17 23:01:18.749702 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.749569    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 23:01:18.749702 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.749569    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 23:01:18.749702 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.749574    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 23:01:18.753274 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.753020    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"]
Sep 17 23:01:18.826782 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.826756    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4063fa31-c500-4826-809f-a9a044db06dc-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.826929 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.826788    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.826929 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.826819    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww7lm\" (UniqueName: \"kubernetes.io/projected/4063fa31-c500-4826-809f-a9a044db06dc-kube-api-access-ww7lm\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.826929 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.826881    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.826929 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.826907    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.827092 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.826934    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.927631 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.927602    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.927843 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.927637    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4063fa31-c500-4826-809f-a9a044db06dc-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.927843 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.927656    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.927843 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.927681    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7lm\" (UniqueName: \"kubernetes.io/projected/4063fa31-c500-4826-809f-a9a044db06dc-kube-api-access-ww7lm\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.927843 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.927699    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.927843 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.927717    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.928141 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.928069    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.928141 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.928099    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.928211 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.928156    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.929947 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.929928    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.930255 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.930235    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4063fa31-c500-4826-809f-a9a044db06dc-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.935189 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.935171    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww7lm\" (UniqueName: \"kubernetes.io/projected/4063fa31-c500-4826-809f-a9a044db06dc-kube-api-access-ww7lm\") pod \"scheduler-inline-config-test-kserve-8655bc757-2wk45\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:18.981280 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.981259    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"]
Sep 17 23:01:18.984304 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.984287    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"
Sep 17 23:01:18.986709 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.986690    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-vj8nk\""
Sep 17 23:01:18.991915 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:18.991862    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"]
Sep 17 23:01:19.056696 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.056674    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:19.129149 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.129110    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53ac1524-4b33-46e4-9710-931c479af59a-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9\" (UID: \"53ac1524-4b33-46e4-9710-931c479af59a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"
Sep 17 23:01:19.129321 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.129288    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfnzz\" (UniqueName: \"kubernetes.io/projected/53ac1524-4b33-46e4-9710-931c479af59a-kube-api-access-qfnzz\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9\" (UID: \"53ac1524-4b33-46e4-9710-931c479af59a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"
Sep 17 23:01:19.229866 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.229839    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53ac1524-4b33-46e4-9710-931c479af59a-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9\" (UID: \"53ac1524-4b33-46e4-9710-931c479af59a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"
Sep 17 23:01:19.230021 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.229885    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfnzz\" (UniqueName: \"kubernetes.io/projected/53ac1524-4b33-46e4-9710-931c479af59a-kube-api-access-qfnzz\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9\" (UID: \"53ac1524-4b33-46e4-9710-931c479af59a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"
Sep 17 23:01:19.232405 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.232383    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53ac1524-4b33-46e4-9710-931c479af59a-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9\" (UID: \"53ac1524-4b33-46e4-9710-931c479af59a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"
Sep 17 23:01:19.237060 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.237042    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfnzz\" (UniqueName: \"kubernetes.io/projected/53ac1524-4b33-46e4-9710-931c479af59a-kube-api-access-qfnzz\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9\" (UID: \"53ac1524-4b33-46e4-9710-931c479af59a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"
Sep 17 23:01:19.294808 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.294743    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"
Sep 17 23:01:19.323100 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.323075    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"]
Sep 17 23:01:19.324613 ip-10-0-142-197 kubenswrapper[2567]: W0917 23:01:19.324592    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4063fa31_c500_4826_809f_a9a044db06dc.slice/crio-76fc68b9a50a705af9e2777d2b2101b9a7ad109f1110a93a7e9ed7a75bd5a018 WatchSource:0}: Error finding container 76fc68b9a50a705af9e2777d2b2101b9a7ad109f1110a93a7e9ed7a75bd5a018: Status 404 returned error can't find the container with id 76fc68b9a50a705af9e2777d2b2101b9a7ad109f1110a93a7e9ed7a75bd5a018
Sep 17 23:01:19.565308 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.565246    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"]
Sep 17 23:01:19.567236 ip-10-0-142-197 kubenswrapper[2567]: W0917 23:01:19.567211    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ac1524_4b33_46e4_9710_931c479af59a.slice/crio-63fc37d6d46339bdf2e259426937c1cf464c19586d3a19cb381f343acb7954c9 WatchSource:0}: Error finding container 63fc37d6d46339bdf2e259426937c1cf464c19586d3a19cb381f343acb7954c9: Status 404 returned error can't find the container with id 63fc37d6d46339bdf2e259426937c1cf464c19586d3a19cb381f343acb7954c9
Sep 17 23:01:19.718816 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.718782    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45" event={"ID":"4063fa31-c500-4826-809f-a9a044db06dc","Type":"ContainerStarted","Data":"32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4"}
Sep 17 23:01:19.719022 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.718841    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45" event={"ID":"4063fa31-c500-4826-809f-a9a044db06dc","Type":"ContainerStarted","Data":"76fc68b9a50a705af9e2777d2b2101b9a7ad109f1110a93a7e9ed7a75bd5a018"}
Sep 17 23:01:19.720459 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.720425    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9" event={"ID":"53ac1524-4b33-46e4-9710-931c479af59a","Type":"ContainerStarted","Data":"11f61bbf4b6b78000175af5f1005223a3d224808135f860f61e2fbf8ea4a73b3"}
Sep 17 23:01:19.720570 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.720475    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9" event={"ID":"53ac1524-4b33-46e4-9710-931c479af59a","Type":"ContainerStarted","Data":"63fc37d6d46339bdf2e259426937c1cf464c19586d3a19cb381f343acb7954c9"}
Sep 17 23:01:19.720570 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.720517    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"
Sep 17 23:01:19.735269 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.735217    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45" podStartSLOduration=1.735201867 podStartE2EDuration="1.735201867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:01:19.733430022 +0000 UTC m=+3756.742359565" watchObservedRunningTime="2026-09-17 23:01:19.735201867 +0000 UTC m=+3756.744131422"
Sep 17 23:01:19.749104 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:19.749036    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9" podStartSLOduration=1.749020249 podStartE2EDuration="1.749020249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:01:19.749020706 +0000 UTC m=+3756.757950248" watchObservedRunningTime="2026-09-17 23:01:19.749020249 +0000 UTC m=+3756.757949789"
Sep 17 23:01:20.057093 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:20.057062    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:29.057805 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:29.057757    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:29.070121 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:29.070099    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:29.772261 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:29.772235    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:50.726817 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:50.726785    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"
Sep 17 23:01:52.401764 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.401727    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"]
Sep 17 23:01:52.410244 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.410214    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"]
Sep 17 23:01:52.421145 ip-10-0-142-197 kubenswrapper[2567]: E0917 23:01:52.421097    2567 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:01:52.421145 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:01:52.421145 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 23:01:52.421145 ip-10-0-142-197 kubenswrapper[2567]:  > containerID="32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4" cmd=["/bin/sleep","15"]
Sep 17 23:01:52.421452 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.421156    2567 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45" message="" err=<
Sep 17 23:01:52.421452 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:01:52.421452 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 23:01:52.421452 ip-10-0-142-197 kubenswrapper[2567]:  >
Sep 17 23:01:52.421452 ip-10-0-142-197 kubenswrapper[2567]: E0917 23:01:52.421173    2567 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:01:52.421452 ip-10-0-142-197 kubenswrapper[2567]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:01:52.421452 ip-10-0-142-197 kubenswrapper[2567]:         , stdout: , stderr: , exit code -1
Sep 17 23:01:52.421452 ip-10-0-142-197 kubenswrapper[2567]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45" podUID="4063fa31-c500-4826-809f-a9a044db06dc" containerName="main" containerID="cri-o://32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4"
Sep 17 23:01:52.421452 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.421232    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45" podUID="4063fa31-c500-4826-809f-a9a044db06dc" containerName="main" containerID="cri-o://32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4" gracePeriod=60
Sep 17 23:01:52.765658 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.765625    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:52.844017 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.843959    2567 generic.go:356] "Generic (PLEG): container finished" podID="4063fa31-c500-4826-809f-a9a044db06dc" containerID="32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4" exitCode=0
Sep 17 23:01:52.844017 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.844010    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"
Sep 17 23:01:52.844236 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.844031    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45" event={"ID":"4063fa31-c500-4826-809f-a9a044db06dc","Type":"ContainerDied","Data":"32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4"}
Sep 17 23:01:52.844236 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.844091    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45" event={"ID":"4063fa31-c500-4826-809f-a9a044db06dc","Type":"ContainerDied","Data":"76fc68b9a50a705af9e2777d2b2101b9a7ad109f1110a93a7e9ed7a75bd5a018"}
Sep 17 23:01:52.844236 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.844122    2567 scope.go:117] "RemoveContainer" containerID="32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4"
Sep 17 23:01:52.858089 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.858069    2567 scope.go:117] "RemoveContainer" containerID="32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4"
Sep 17 23:01:52.858386 ip-10-0-142-197 kubenswrapper[2567]: E0917 23:01:52.858364    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4\": container with ID starting with 32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4 not found: ID does not exist" containerID="32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4"
Sep 17 23:01:52.858449 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.858396    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4"} err="failed to get container status \"32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4\": rpc error: code = NotFound desc = could not find container \"32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4\": container with ID starting with 32c0785c5b8fba8e0a59ef3615e908e699bf43fa8019eb05e3b8eac672174bf4 not found: ID does not exist"
Sep 17 23:01:52.866661 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.866641    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4063fa31-c500-4826-809f-a9a044db06dc-tls-certs\") pod \"4063fa31-c500-4826-809f-a9a044db06dc\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") "
Sep 17 23:01:52.866772 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.866682    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-dshm\") pod \"4063fa31-c500-4826-809f-a9a044db06dc\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") "
Sep 17 23:01:52.866772 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.866701    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-model-cache\") pod \"4063fa31-c500-4826-809f-a9a044db06dc\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") "
Sep 17 23:01:52.866772 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.866726    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-tmp-dir\") pod \"4063fa31-c500-4826-809f-a9a044db06dc\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") "
Sep 17 23:01:52.866937 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.866787    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-home\") pod \"4063fa31-c500-4826-809f-a9a044db06dc\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") "
Sep 17 23:01:52.866937 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.866818    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww7lm\" (UniqueName: \"kubernetes.io/projected/4063fa31-c500-4826-809f-a9a044db06dc-kube-api-access-ww7lm\") pod \"4063fa31-c500-4826-809f-a9a044db06dc\" (UID: \"4063fa31-c500-4826-809f-a9a044db06dc\") "
Sep 17 23:01:52.867065 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.866953    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4063fa31-c500-4826-809f-a9a044db06dc" (UID: "4063fa31-c500-4826-809f-a9a044db06dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:01:52.867065 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.866965    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "4063fa31-c500-4826-809f-a9a044db06dc" (UID: "4063fa31-c500-4826-809f-a9a044db06dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:01:52.867065 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.867046    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-home" (OuterVolumeSpecName: "home") pod "4063fa31-c500-4826-809f-a9a044db06dc" (UID: "4063fa31-c500-4826-809f-a9a044db06dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:01:52.867178 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.867093    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:01:52.867178 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.867106    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:01:52.868856 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.868833    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-dshm" (OuterVolumeSpecName: "dshm") pod "4063fa31-c500-4826-809f-a9a044db06dc" (UID: "4063fa31-c500-4826-809f-a9a044db06dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:01:52.868951 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.868898    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4063fa31-c500-4826-809f-a9a044db06dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4063fa31-c500-4826-809f-a9a044db06dc" (UID: "4063fa31-c500-4826-809f-a9a044db06dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:01:52.868951 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.868912    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4063fa31-c500-4826-809f-a9a044db06dc-kube-api-access-ww7lm" (OuterVolumeSpecName: "kube-api-access-ww7lm") pod "4063fa31-c500-4826-809f-a9a044db06dc" (UID: "4063fa31-c500-4826-809f-a9a044db06dc"). InnerVolumeSpecName "kube-api-access-ww7lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:01:52.968016 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.967941    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:01:52.968016 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.967963    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4063fa31-c500-4826-809f-a9a044db06dc-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:01:52.968016 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.967971    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww7lm\" (UniqueName: \"kubernetes.io/projected/4063fa31-c500-4826-809f-a9a044db06dc-kube-api-access-ww7lm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:01:52.968016 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:52.967981    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4063fa31-c500-4826-809f-a9a044db06dc-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:01:53.164972 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.164946    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"]
Sep 17 23:01:53.168411 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.168387    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45"]
Sep 17 23:01:53.639375 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.639338    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4063fa31-c500-4826-809f-a9a044db06dc" path="/var/lib/kubelet/pods/4063fa31-c500-4826-809f-a9a044db06dc/volumes"
Sep 17 23:01:53.737646 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.737620    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"]
Sep 17 23:01:53.737865 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.737854    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4063fa31-c500-4826-809f-a9a044db06dc" containerName="main"
Sep 17 23:01:53.737908 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.737867    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4063fa31-c500-4826-809f-a9a044db06dc" containerName="main"
Sep 17 23:01:53.737945 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.737922    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4063fa31-c500-4826-809f-a9a044db06dc" containerName="main"
Sep 17 23:01:53.742038 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.742022    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.744272 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.744228    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 23:01:53.744272 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.744242    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 23:01:53.744272 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.744226    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 23:01:53.744448 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.744400    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-fg582\""
Sep 17 23:01:53.744558 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.744541    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:01:53.744615 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.744588    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 23:01:53.754529 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.754510    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"]
Sep 17 23:01:53.762649 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.762624    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2wk45" podUID="4063fa31-c500-4826-809f-a9a044db06dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Sep 17 23:01:53.873392 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.873368    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e8b17d-8c60-45f1-8e4d-b7720249cefa-tls-certs\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.873523 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.873401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96e8b17d-8c60-45f1-8e4d-b7720249cefa-cabundle-cert\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.873523 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.873429    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-tmp-dir\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.873523 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.873451    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmvkp\" (UniqueName: \"kubernetes.io/projected/96e8b17d-8c60-45f1-8e4d-b7720249cefa-kube-api-access-fmvkp\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.873523 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.873514    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-dshm\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.873662 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.873538    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-model-cache\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.873662 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.873556    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-home\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.873662 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.873593    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-kserve-provision-location\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.974420 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.974357    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-dshm\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.974420 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.974387    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-model-cache\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.974420 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.974404    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-home\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.974644 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.974438    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-kserve-provision-location\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.974644 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.974550    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e8b17d-8c60-45f1-8e4d-b7720249cefa-tls-certs\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.974644 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.974586    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96e8b17d-8c60-45f1-8e4d-b7720249cefa-cabundle-cert\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.974644 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.974612    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-tmp-dir\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.974644 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.974629    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmvkp\" (UniqueName: \"kubernetes.io/projected/96e8b17d-8c60-45f1-8e4d-b7720249cefa-kube-api-access-fmvkp\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.974892 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.974863    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-kserve-provision-location\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.974951 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.974902    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-model-cache\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.975024 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.974952    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-tmp-dir\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.975024 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.974960    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-home\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.975452 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.975432    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96e8b17d-8c60-45f1-8e4d-b7720249cefa-cabundle-cert\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.976713 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.976682    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-dshm\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.976953 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.976938    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e8b17d-8c60-45f1-8e4d-b7720249cefa-tls-certs\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:53.982852 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:53.982832    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmvkp\" (UniqueName: \"kubernetes.io/projected/96e8b17d-8c60-45f1-8e4d-b7720249cefa-kube-api-access-fmvkp\") pod \"prestop-hook-test-kserve-5bcc844858-h82x9\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:54.051721 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:54.051698    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:01:54.325417 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:54.325389    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"]
Sep 17 23:01:54.326159 ip-10-0-142-197 kubenswrapper[2567]: W0917 23:01:54.326135    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e8b17d_8c60_45f1_8e4d_b7720249cefa.slice/crio-cd657ccb482b6b4f47f004e9d6047ee6a8e7f7fef3b9fc24f6df86cbf7f6d45f WatchSource:0}: Error finding container cd657ccb482b6b4f47f004e9d6047ee6a8e7f7fef3b9fc24f6df86cbf7f6d45f: Status 404 returned error can't find the container with id cd657ccb482b6b4f47f004e9d6047ee6a8e7f7fef3b9fc24f6df86cbf7f6d45f
Sep 17 23:01:54.854168 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:54.854126    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9" event={"ID":"96e8b17d-8c60-45f1-8e4d-b7720249cefa","Type":"ContainerStarted","Data":"56dfa18027fba589c8562b69582d813d0d7fe2500ca4fb5dc26843526b618242"}
Sep 17 23:01:54.854579 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:54.854186    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9" event={"ID":"96e8b17d-8c60-45f1-8e4d-b7720249cefa","Type":"ContainerStarted","Data":"cd657ccb482b6b4f47f004e9d6047ee6a8e7f7fef3b9fc24f6df86cbf7f6d45f"}
Sep 17 23:01:56.866043 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:56.865986    2567 generic.go:356] "Generic (PLEG): container finished" podID="96e8b17d-8c60-45f1-8e4d-b7720249cefa" containerID="56dfa18027fba589c8562b69582d813d0d7fe2500ca4fb5dc26843526b618242" exitCode=0
Sep 17 23:01:56.866417 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:56.866065    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9" event={"ID":"96e8b17d-8c60-45f1-8e4d-b7720249cefa","Type":"ContainerDied","Data":"56dfa18027fba589c8562b69582d813d0d7fe2500ca4fb5dc26843526b618242"}
Sep 17 23:01:57.872103 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:57.872059    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9" event={"ID":"96e8b17d-8c60-45f1-8e4d-b7720249cefa","Type":"ContainerStarted","Data":"2f0d32c1099879aafee3f885e754b501e424c62fd96e1a4047af334b30c14025"}
Sep 17 23:01:57.894152 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:57.894103    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9" podStartSLOduration=2.8737969679999997 podStartE2EDuration="4.894088237s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.020291269s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:01:57.892976203 +0000 UTC m=+3794.901905744" watchObservedRunningTime="2026-09-17 23:01:57.894088237 +0000 UTC m=+3794.903017795"
Sep 17 23:01:58.052613 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:01:58.052574    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:02:00.726660 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:00.726615    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9" podUID="53ac1524-4b33-46e4-9710-931c479af59a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:02:04.052482 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:04.052445    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:02:04.053986 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:04.053959    2567 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9" podUID="96e8b17d-8c60-45f1-8e4d-b7720249cefa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 17 23:02:07.424934 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:07.424893    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9" podUID="53ac1524-4b33-46e4-9710-931c479af59a" containerName="main" containerID="cri-o://11f61bbf4b6b78000175af5f1005223a3d224808135f860f61e2fbf8ea4a73b3" gracePeriod=45
Sep 17 23:02:10.726660 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:10.726623    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9" podUID="53ac1524-4b33-46e4-9710-931c479af59a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:02:12.089560 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:12.089526    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"]
Sep 17 23:02:20.726826 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:20.726784    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9" podUID="53ac1524-4b33-46e4-9710-931c479af59a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:02:20.727330 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:20.726879    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"
Sep 17 23:02:27.110952 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:27.110913    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9" podUID="96e8b17d-8c60-45f1-8e4d-b7720249cefa" containerName="main" containerID="cri-o://2f0d32c1099879aafee3f885e754b501e424c62fd96e1a4047af334b30c14025" gracePeriod=45
Sep 17 23:02:30.727308 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:30.727273    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9" podUID="53ac1524-4b33-46e4-9710-931c479af59a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:02:37.661876 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:37.661845    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"
Sep 17 23:02:37.815072 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:37.815032    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53ac1524-4b33-46e4-9710-931c479af59a-tls-certs\") pod \"53ac1524-4b33-46e4-9710-931c479af59a\" (UID: \"53ac1524-4b33-46e4-9710-931c479af59a\") "
Sep 17 23:02:37.815262 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:37.815083    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfnzz\" (UniqueName: \"kubernetes.io/projected/53ac1524-4b33-46e4-9710-931c479af59a-kube-api-access-qfnzz\") pod \"53ac1524-4b33-46e4-9710-931c479af59a\" (UID: \"53ac1524-4b33-46e4-9710-931c479af59a\") "
Sep 17 23:02:37.817221 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:37.817181    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ac1524-4b33-46e4-9710-931c479af59a-kube-api-access-qfnzz" (OuterVolumeSpecName: "kube-api-access-qfnzz") pod "53ac1524-4b33-46e4-9710-931c479af59a" (UID: "53ac1524-4b33-46e4-9710-931c479af59a"). InnerVolumeSpecName "kube-api-access-qfnzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:02:37.817333 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:37.817245    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ac1524-4b33-46e4-9710-931c479af59a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53ac1524-4b33-46e4-9710-931c479af59a" (UID: "53ac1524-4b33-46e4-9710-931c479af59a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:02:37.915633 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:37.915604    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53ac1524-4b33-46e4-9710-931c479af59a-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:02:37.915633 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:37.915630    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfnzz\" (UniqueName: \"kubernetes.io/projected/53ac1524-4b33-46e4-9710-931c479af59a-kube-api-access-qfnzz\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:02:38.029619 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:38.029585    2567 generic.go:356] "Generic (PLEG): container finished" podID="53ac1524-4b33-46e4-9710-931c479af59a" containerID="11f61bbf4b6b78000175af5f1005223a3d224808135f860f61e2fbf8ea4a73b3" exitCode=0
Sep 17 23:02:38.029819 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:38.029644    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"
Sep 17 23:02:38.029819 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:38.029676    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9" event={"ID":"53ac1524-4b33-46e4-9710-931c479af59a","Type":"ContainerDied","Data":"11f61bbf4b6b78000175af5f1005223a3d224808135f860f61e2fbf8ea4a73b3"}
Sep 17 23:02:38.029819 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:38.029739    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9" event={"ID":"53ac1524-4b33-46e4-9710-931c479af59a","Type":"ContainerDied","Data":"63fc37d6d46339bdf2e259426937c1cf464c19586d3a19cb381f343acb7954c9"}
Sep 17 23:02:38.029819 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:38.029769    2567 scope.go:117] "RemoveContainer" containerID="11f61bbf4b6b78000175af5f1005223a3d224808135f860f61e2fbf8ea4a73b3"
Sep 17 23:02:38.037822 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:38.037804    2567 scope.go:117] "RemoveContainer" containerID="11f61bbf4b6b78000175af5f1005223a3d224808135f860f61e2fbf8ea4a73b3"
Sep 17 23:02:38.038118 ip-10-0-142-197 kubenswrapper[2567]: E0917 23:02:38.038098    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f61bbf4b6b78000175af5f1005223a3d224808135f860f61e2fbf8ea4a73b3\": container with ID starting with 11f61bbf4b6b78000175af5f1005223a3d224808135f860f61e2fbf8ea4a73b3 not found: ID does not exist" containerID="11f61bbf4b6b78000175af5f1005223a3d224808135f860f61e2fbf8ea4a73b3"
Sep 17 23:02:38.038203 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:38.038128    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f61bbf4b6b78000175af5f1005223a3d224808135f860f61e2fbf8ea4a73b3"} err="failed to get container status \"11f61bbf4b6b78000175af5f1005223a3d224808135f860f61e2fbf8ea4a73b3\": rpc error: code = NotFound desc = could not find container \"11f61bbf4b6b78000175af5f1005223a3d224808135f860f61e2fbf8ea4a73b3\": container with ID starting with 11f61bbf4b6b78000175af5f1005223a3d224808135f860f61e2fbf8ea4a73b3 not found: ID does not exist"
Sep 17 23:02:38.052487 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:38.052462    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"]
Sep 17 23:02:38.055600 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:38.055580    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbg58b9"]
Sep 17 23:02:39.639523 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:39.639480    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53ac1524-4b33-46e4-9710-931c479af59a" path="/var/lib/kubelet/pods/53ac1524-4b33-46e4-9710-931c479af59a/volumes"
Sep 17 23:02:47.432986 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.432944    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"]
Sep 17 23:02:47.433435 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.433253    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ac1524-4b33-46e4-9710-931c479af59a" containerName="main"
Sep 17 23:02:47.433435 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.433265    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ac1524-4b33-46e4-9710-931c479af59a" containerName="main"
Sep 17 23:02:47.433435 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.433312    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="53ac1524-4b33-46e4-9710-931c479af59a" containerName="main"
Sep 17 23:02:47.435915 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.435897    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"
Sep 17 23:02:47.438260 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.438232    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 23:02:47.438408 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.438261    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 23:02:47.439328 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.439288    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 23:02:47.439447 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.439357    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-nchkf\""
Sep 17 23:02:47.439447 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.439315    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-tj6xs\""
Sep 17 23:02:47.445091 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.445067    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"]
Sep 17 23:02:47.591894 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.591852    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frq4f\" (UniqueName: \"kubernetes.io/projected/2944c60f-7cf1-4fd0-a276-3bc616a0d8b5-kube-api-access-frq4f\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg\" (UID: \"2944c60f-7cf1-4fd0-a276-3bc616a0d8b5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"
Sep 17 23:02:47.592130 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.591912    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2944c60f-7cf1-4fd0-a276-3bc616a0d8b5-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg\" (UID: \"2944c60f-7cf1-4fd0-a276-3bc616a0d8b5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"
Sep 17 23:02:47.693158 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.693069    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2944c60f-7cf1-4fd0-a276-3bc616a0d8b5-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg\" (UID: \"2944c60f-7cf1-4fd0-a276-3bc616a0d8b5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"
Sep 17 23:02:47.693158 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.693134    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frq4f\" (UniqueName: \"kubernetes.io/projected/2944c60f-7cf1-4fd0-a276-3bc616a0d8b5-kube-api-access-frq4f\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg\" (UID: \"2944c60f-7cf1-4fd0-a276-3bc616a0d8b5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"
Sep 17 23:02:47.695576 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.695552    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2944c60f-7cf1-4fd0-a276-3bc616a0d8b5-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg\" (UID: \"2944c60f-7cf1-4fd0-a276-3bc616a0d8b5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"
Sep 17 23:02:47.705141 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.705113    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frq4f\" (UniqueName: \"kubernetes.io/projected/2944c60f-7cf1-4fd0-a276-3bc616a0d8b5-kube-api-access-frq4f\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg\" (UID: \"2944c60f-7cf1-4fd0-a276-3bc616a0d8b5\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"
Sep 17 23:02:47.746941 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:47.746911    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"
Sep 17 23:02:48.029342 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:48.029310    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"]
Sep 17 23:02:48.033142 ip-10-0-142-197 kubenswrapper[2567]: W0917 23:02:48.033115    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2944c60f_7cf1_4fd0_a276_3bc616a0d8b5.slice/crio-5e71856eed24b27c51d8d2a402f40ce71521f628a0ec9fe842a50bc0ae161d4d WatchSource:0}: Error finding container 5e71856eed24b27c51d8d2a402f40ce71521f628a0ec9fe842a50bc0ae161d4d: Status 404 returned error can't find the container with id 5e71856eed24b27c51d8d2a402f40ce71521f628a0ec9fe842a50bc0ae161d4d
Sep 17 23:02:48.068119 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:48.068083    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg" event={"ID":"2944c60f-7cf1-4fd0-a276-3bc616a0d8b5","Type":"ContainerStarted","Data":"5e71856eed24b27c51d8d2a402f40ce71521f628a0ec9fe842a50bc0ae161d4d"}
Sep 17 23:02:49.073500 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:49.073463    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg" event={"ID":"2944c60f-7cf1-4fd0-a276-3bc616a0d8b5","Type":"ContainerStarted","Data":"19ee6a7880eada2a47a8c10d088e218e8a401d2fe2c72afc6b9708933a7bdba6"}
Sep 17 23:02:49.073967 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:49.073571    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"
Sep 17 23:02:49.089720 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:02:49.089667    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg" podStartSLOduration=2.089651482 podStartE2EDuration="2.089651482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:02:49.088988109 +0000 UTC m=+3846.097917650" watchObservedRunningTime="2026-09-17 23:02:49.089651482 +0000 UTC m=+3846.098581024"
Sep 17 23:03:12.752741 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.752713    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-5bcc844858-h82x9_96e8b17d-8c60-45f1-8e4d-b7720249cefa/main/0.log"
Sep 17 23:03:12.754291 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.754274    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:03:12.785036 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.784989    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e8b17d-8c60-45f1-8e4d-b7720249cefa-tls-certs\") pod \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") "
Sep 17 23:03:12.785182 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.785053    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-kserve-provision-location\") pod \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") "
Sep 17 23:03:12.785182 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.785098    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96e8b17d-8c60-45f1-8e4d-b7720249cefa-cabundle-cert\") pod \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") "
Sep 17 23:03:12.785182 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.785139    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-model-cache\") pod \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") "
Sep 17 23:03:12.785376 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.785191    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmvkp\" (UniqueName: \"kubernetes.io/projected/96e8b17d-8c60-45f1-8e4d-b7720249cefa-kube-api-access-fmvkp\") pod \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") "
Sep 17 23:03:12.785376 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.785225    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-tmp-dir\") pod \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") "
Sep 17 23:03:12.785376 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.785266    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-dshm\") pod \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") "
Sep 17 23:03:12.785376 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.785291    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-home\") pod \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\" (UID: \"96e8b17d-8c60-45f1-8e4d-b7720249cefa\") "
Sep 17 23:03:12.785571 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.785455    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-model-cache" (OuterVolumeSpecName: "model-cache") pod "96e8b17d-8c60-45f1-8e4d-b7720249cefa" (UID: "96e8b17d-8c60-45f1-8e4d-b7720249cefa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:03:12.785571 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.785510    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e8b17d-8c60-45f1-8e4d-b7720249cefa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "96e8b17d-8c60-45f1-8e4d-b7720249cefa" (UID: "96e8b17d-8c60-45f1-8e4d-b7720249cefa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:03:12.785571 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.785545    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-model-cache\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:03:12.785868 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.785838    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-home" (OuterVolumeSpecName: "home") pod "96e8b17d-8c60-45f1-8e4d-b7720249cefa" (UID: "96e8b17d-8c60-45f1-8e4d-b7720249cefa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:03:12.786047 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.786022    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96e8b17d-8c60-45f1-8e4d-b7720249cefa" (UID: "96e8b17d-8c60-45f1-8e4d-b7720249cefa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:03:12.787680 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.787650    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e8b17d-8c60-45f1-8e4d-b7720249cefa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96e8b17d-8c60-45f1-8e4d-b7720249cefa" (UID: "96e8b17d-8c60-45f1-8e4d-b7720249cefa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:03:12.787804 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.787781    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-dshm" (OuterVolumeSpecName: "dshm") pod "96e8b17d-8c60-45f1-8e4d-b7720249cefa" (UID: "96e8b17d-8c60-45f1-8e4d-b7720249cefa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:03:12.787912 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.787812    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e8b17d-8c60-45f1-8e4d-b7720249cefa-kube-api-access-fmvkp" (OuterVolumeSpecName: "kube-api-access-fmvkp") pod "96e8b17d-8c60-45f1-8e4d-b7720249cefa" (UID: "96e8b17d-8c60-45f1-8e4d-b7720249cefa"). InnerVolumeSpecName "kube-api-access-fmvkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:03:12.808026 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.807983    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96e8b17d-8c60-45f1-8e4d-b7720249cefa" (UID: "96e8b17d-8c60-45f1-8e4d-b7720249cefa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:03:12.886286 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.886217    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmvkp\" (UniqueName: \"kubernetes.io/projected/96e8b17d-8c60-45f1-8e4d-b7720249cefa-kube-api-access-fmvkp\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:03:12.886286 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.886244    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-tmp-dir\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:03:12.886286 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.886253    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-dshm\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:03:12.886286 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.886261    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-home\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:03:12.886286 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.886268    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e8b17d-8c60-45f1-8e4d-b7720249cefa-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:03:12.886286 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.886277    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96e8b17d-8c60-45f1-8e4d-b7720249cefa-kserve-provision-location\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:03:12.886286 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:12.886284    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96e8b17d-8c60-45f1-8e4d-b7720249cefa-cabundle-cert\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:03:13.165475 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:13.165398    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-5bcc844858-h82x9_96e8b17d-8c60-45f1-8e4d-b7720249cefa/main/0.log"
Sep 17 23:03:13.165803 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:13.165781    2567 generic.go:356] "Generic (PLEG): container finished" podID="96e8b17d-8c60-45f1-8e4d-b7720249cefa" containerID="2f0d32c1099879aafee3f885e754b501e424c62fd96e1a4047af334b30c14025" exitCode=137
Sep 17 23:03:13.165853 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:13.165834    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"
Sep 17 23:03:13.165889 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:13.165851    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9" event={"ID":"96e8b17d-8c60-45f1-8e4d-b7720249cefa","Type":"ContainerDied","Data":"2f0d32c1099879aafee3f885e754b501e424c62fd96e1a4047af334b30c14025"}
Sep 17 23:03:13.165928 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:13.165892    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9" event={"ID":"96e8b17d-8c60-45f1-8e4d-b7720249cefa","Type":"ContainerDied","Data":"cd657ccb482b6b4f47f004e9d6047ee6a8e7f7fef3b9fc24f6df86cbf7f6d45f"}
Sep 17 23:03:13.165928 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:13.165913    2567 scope.go:117] "RemoveContainer" containerID="2f0d32c1099879aafee3f885e754b501e424c62fd96e1a4047af334b30c14025"
Sep 17 23:03:13.174476 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:13.174427    2567 scope.go:117] "RemoveContainer" containerID="56dfa18027fba589c8562b69582d813d0d7fe2500ca4fb5dc26843526b618242"
Sep 17 23:03:13.187686 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:13.187435    2567 scope.go:117] "RemoveContainer" containerID="2f0d32c1099879aafee3f885e754b501e424c62fd96e1a4047af334b30c14025"
Sep 17 23:03:13.187905 ip-10-0-142-197 kubenswrapper[2567]: E0917 23:03:13.187871    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0d32c1099879aafee3f885e754b501e424c62fd96e1a4047af334b30c14025\": container with ID starting with 2f0d32c1099879aafee3f885e754b501e424c62fd96e1a4047af334b30c14025 not found: ID does not exist" containerID="2f0d32c1099879aafee3f885e754b501e424c62fd96e1a4047af334b30c14025"
Sep 17 23:03:13.188021 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:13.187920    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0d32c1099879aafee3f885e754b501e424c62fd96e1a4047af334b30c14025"} err="failed to get container status \"2f0d32c1099879aafee3f885e754b501e424c62fd96e1a4047af334b30c14025\": rpc error: code = NotFound desc = could not find container \"2f0d32c1099879aafee3f885e754b501e424c62fd96e1a4047af334b30c14025\": container with ID starting with 2f0d32c1099879aafee3f885e754b501e424c62fd96e1a4047af334b30c14025 not found: ID does not exist"
Sep 17 23:03:13.188021 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:13.187947    2567 scope.go:117] "RemoveContainer" containerID="56dfa18027fba589c8562b69582d813d0d7fe2500ca4fb5dc26843526b618242"
Sep 17 23:03:13.188353 ip-10-0-142-197 kubenswrapper[2567]: E0917 23:03:13.188307    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56dfa18027fba589c8562b69582d813d0d7fe2500ca4fb5dc26843526b618242\": container with ID starting with 56dfa18027fba589c8562b69582d813d0d7fe2500ca4fb5dc26843526b618242 not found: ID does not exist" containerID="56dfa18027fba589c8562b69582d813d0d7fe2500ca4fb5dc26843526b618242"
Sep 17 23:03:13.188353 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:13.188338    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56dfa18027fba589c8562b69582d813d0d7fe2500ca4fb5dc26843526b618242"} err="failed to get container status \"56dfa18027fba589c8562b69582d813d0d7fe2500ca4fb5dc26843526b618242\": rpc error: code = NotFound desc = could not find container \"56dfa18027fba589c8562b69582d813d0d7fe2500ca4fb5dc26843526b618242\": container with ID starting with 56dfa18027fba589c8562b69582d813d0d7fe2500ca4fb5dc26843526b618242 not found: ID does not exist"
Sep 17 23:03:13.188457 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:13.188398    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"]
Sep 17 23:03:13.191507 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:13.191486    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5bcc844858-h82x9"]
Sep 17 23:03:13.639924 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:13.639897    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96e8b17d-8c60-45f1-8e4d-b7720249cefa" path="/var/lib/kubelet/pods/96e8b17d-8c60-45f1-8e4d-b7720249cefa/volumes"
Sep 17 23:03:20.079218 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:20.079192    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"
Sep 17 23:03:26.382573 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.382531    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"]
Sep 17 23:03:26.382963 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.382801    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96e8b17d-8c60-45f1-8e4d-b7720249cefa" containerName="storage-initializer"
Sep 17 23:03:26.382963 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.382813    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="96e8b17d-8c60-45f1-8e4d-b7720249cefa" containerName="storage-initializer"
Sep 17 23:03:26.382963 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.382831    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96e8b17d-8c60-45f1-8e4d-b7720249cefa" containerName="main"
Sep 17 23:03:26.382963 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.382837    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="96e8b17d-8c60-45f1-8e4d-b7720249cefa" containerName="main"
Sep 17 23:03:26.382963 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.382882    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="96e8b17d-8c60-45f1-8e4d-b7720249cefa" containerName="main"
Sep 17 23:03:26.385562 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.385546    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"
Sep 17 23:03:26.391281 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.391252    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 23:03:26.391281 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.391277    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 23:03:26.391489 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.391409    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-pmr2x\""
Sep 17 23:03:26.391553 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.391511    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-d9rk5\""
Sep 17 23:03:26.391613 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.391568    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 23:03:26.400043 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.400021    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"]
Sep 17 23:03:26.481647 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.481606    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845\" (UID: \"5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"
Sep 17 23:03:26.481821 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.481653    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzlgh\" (UniqueName: \"kubernetes.io/projected/5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d-kube-api-access-dzlgh\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845\" (UID: \"5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"
Sep 17 23:03:26.582573 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.582538    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845\" (UID: \"5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"
Sep 17 23:03:26.582573 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.582575    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzlgh\" (UniqueName: \"kubernetes.io/projected/5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d-kube-api-access-dzlgh\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845\" (UID: \"5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"
Sep 17 23:03:26.585015 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.584961    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845\" (UID: \"5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"
Sep 17 23:03:26.591827 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.591805    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzlgh\" (UniqueName: \"kubernetes.io/projected/5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d-kube-api-access-dzlgh\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845\" (UID: \"5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"
Sep 17 23:03:26.695127 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.695068    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"
Sep 17 23:03:26.961796 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:26.961715    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"]
Sep 17 23:03:26.966244 ip-10-0-142-197 kubenswrapper[2567]: W0917 23:03:26.966214    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5810dd9a_5cb4_4b8a_9b47_27dd64c9d77d.slice/crio-21fd2fcd080f414acf2bda80b224f4706ec815a91445f8ddc15d5f60061cef72 WatchSource:0}: Error finding container 21fd2fcd080f414acf2bda80b224f4706ec815a91445f8ddc15d5f60061cef72: Status 404 returned error can't find the container with id 21fd2fcd080f414acf2bda80b224f4706ec815a91445f8ddc15d5f60061cef72
Sep 17 23:03:27.219532 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:27.219434    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845" event={"ID":"5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d","Type":"ContainerStarted","Data":"5d20db50a0e81ef106bb0d94303b4e98040fb53d0b23c30b2392e0a013992e1a"}
Sep 17 23:03:27.219532 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:27.219494    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845" event={"ID":"5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d","Type":"ContainerStarted","Data":"21fd2fcd080f414acf2bda80b224f4706ec815a91445f8ddc15d5f60061cef72"}
Sep 17 23:03:27.219758 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:27.219552    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"
Sep 17 23:03:27.235017 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:27.234937    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845" podStartSLOduration=1.234924318 podStartE2EDuration="1.234924318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:03:27.233741963 +0000 UTC m=+3884.242671506" watchObservedRunningTime="2026-09-17 23:03:27.234924318 +0000 UTC m=+3884.243853858"
Sep 17 23:03:37.366139 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:37.366099    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"]
Sep 17 23:03:40.079778 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:40.079743    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg" podUID="2944c60f-7cf1-4fd0-a276-3bc616a0d8b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:03:50.079547 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:50.079505    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg" podUID="2944c60f-7cf1-4fd0-a276-3bc616a0d8b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:03:52.388104 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:52.388053    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg" podUID="2944c60f-7cf1-4fd0-a276-3bc616a0d8b5" containerName="main" containerID="cri-o://19ee6a7880eada2a47a8c10d088e218e8a401d2fe2c72afc6b9708933a7bdba6" gracePeriod=45
Sep 17 23:03:58.224919 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:03:58.224889    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"
Sep 17 23:04:00.079782 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:00.079748    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg" podUID="2944c60f-7cf1-4fd0-a276-3bc616a0d8b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:04:00.080234 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:00.079838    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"
Sep 17 23:04:10.079467 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:10.079430    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg" podUID="2944c60f-7cf1-4fd0-a276-3bc616a0d8b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:04:13.058830 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:13.058790    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"]
Sep 17 23:04:18.225524 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:18.225488    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845" podUID="5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:04:20.079676 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:20.079643    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg" podUID="2944c60f-7cf1-4fd0-a276-3bc616a0d8b5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:04:22.615142 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:22.615116    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"
Sep 17 23:04:22.773880 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:22.773839    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frq4f\" (UniqueName: \"kubernetes.io/projected/2944c60f-7cf1-4fd0-a276-3bc616a0d8b5-kube-api-access-frq4f\") pod \"2944c60f-7cf1-4fd0-a276-3bc616a0d8b5\" (UID: \"2944c60f-7cf1-4fd0-a276-3bc616a0d8b5\") "
Sep 17 23:04:22.773880 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:22.773885    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2944c60f-7cf1-4fd0-a276-3bc616a0d8b5-tls-certs\") pod \"2944c60f-7cf1-4fd0-a276-3bc616a0d8b5\" (UID: \"2944c60f-7cf1-4fd0-a276-3bc616a0d8b5\") "
Sep 17 23:04:22.776044 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:22.775988    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2944c60f-7cf1-4fd0-a276-3bc616a0d8b5-kube-api-access-frq4f" (OuterVolumeSpecName: "kube-api-access-frq4f") pod "2944c60f-7cf1-4fd0-a276-3bc616a0d8b5" (UID: "2944c60f-7cf1-4fd0-a276-3bc616a0d8b5"). InnerVolumeSpecName "kube-api-access-frq4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:04:22.776044 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:22.776027    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2944c60f-7cf1-4fd0-a276-3bc616a0d8b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2944c60f-7cf1-4fd0-a276-3bc616a0d8b5" (UID: "2944c60f-7cf1-4fd0-a276-3bc616a0d8b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:04:22.874474 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:22.874447    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frq4f\" (UniqueName: \"kubernetes.io/projected/2944c60f-7cf1-4fd0-a276-3bc616a0d8b5-kube-api-access-frq4f\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:04:22.874474 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:22.874469    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2944c60f-7cf1-4fd0-a276-3bc616a0d8b5-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:04:23.416091 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:23.416056    2567 generic.go:356] "Generic (PLEG): container finished" podID="2944c60f-7cf1-4fd0-a276-3bc616a0d8b5" containerID="19ee6a7880eada2a47a8c10d088e218e8a401d2fe2c72afc6b9708933a7bdba6" exitCode=0
Sep 17 23:04:23.416254 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:23.416095    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"
Sep 17 23:04:23.416254 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:23.416099    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg" event={"ID":"2944c60f-7cf1-4fd0-a276-3bc616a0d8b5","Type":"ContainerDied","Data":"19ee6a7880eada2a47a8c10d088e218e8a401d2fe2c72afc6b9708933a7bdba6"}
Sep 17 23:04:23.416254 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:23.416226    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg" event={"ID":"2944c60f-7cf1-4fd0-a276-3bc616a0d8b5","Type":"ContainerDied","Data":"5e71856eed24b27c51d8d2a402f40ce71521f628a0ec9fe842a50bc0ae161d4d"}
Sep 17 23:04:23.416363 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:23.416256    2567 scope.go:117] "RemoveContainer" containerID="19ee6a7880eada2a47a8c10d088e218e8a401d2fe2c72afc6b9708933a7bdba6"
Sep 17 23:04:23.424925 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:23.424907    2567 scope.go:117] "RemoveContainer" containerID="19ee6a7880eada2a47a8c10d088e218e8a401d2fe2c72afc6b9708933a7bdba6"
Sep 17 23:04:23.425193 ip-10-0-142-197 kubenswrapper[2567]: E0917 23:04:23.425171    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ee6a7880eada2a47a8c10d088e218e8a401d2fe2c72afc6b9708933a7bdba6\": container with ID starting with 19ee6a7880eada2a47a8c10d088e218e8a401d2fe2c72afc6b9708933a7bdba6 not found: ID does not exist" containerID="19ee6a7880eada2a47a8c10d088e218e8a401d2fe2c72afc6b9708933a7bdba6"
Sep 17 23:04:23.425275 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:23.425201    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ee6a7880eada2a47a8c10d088e218e8a401d2fe2c72afc6b9708933a7bdba6"} err="failed to get container status \"19ee6a7880eada2a47a8c10d088e218e8a401d2fe2c72afc6b9708933a7bdba6\": rpc error: code = NotFound desc = could not find container \"19ee6a7880eada2a47a8c10d088e218e8a401d2fe2c72afc6b9708933a7bdba6\": container with ID starting with 19ee6a7880eada2a47a8c10d088e218e8a401d2fe2c72afc6b9708933a7bdba6 not found: ID does not exist"
Sep 17 23:04:23.436076 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:23.436052    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"]
Sep 17 23:04:23.439187 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:23.439168    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schehsgtg"]
Sep 17 23:04:23.640417 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:23.640380    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2944c60f-7cf1-4fd0-a276-3bc616a0d8b5" path="/var/lib/kubelet/pods/2944c60f-7cf1-4fd0-a276-3bc616a0d8b5/volumes"
Sep 17 23:04:28.077590 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:28.077549    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845" podUID="5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d" containerName="main" containerID="cri-o://5d20db50a0e81ef106bb0d94303b4e98040fb53d0b23c30b2392e0a013992e1a" gracePeriod=45
Sep 17 23:04:28.225316 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:28.225280    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845" podUID="5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:04:38.225170 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:38.225081    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845" podUID="5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:04:38.225712 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:38.225204    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"
Sep 17 23:04:39.039980 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.039947    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"]
Sep 17 23:04:39.040250 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.040238    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2944c60f-7cf1-4fd0-a276-3bc616a0d8b5" containerName="main"
Sep 17 23:04:39.040294 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.040252    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2944c60f-7cf1-4fd0-a276-3bc616a0d8b5" containerName="main"
Sep 17 23:04:39.040329 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.040303    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2944c60f-7cf1-4fd0-a276-3bc616a0d8b5" containerName="main"
Sep 17 23:04:39.042180 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.042163    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"
Sep 17 23:04:39.044598 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.044574    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 23:04:39.045540 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.045518    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-s9hm7\""
Sep 17 23:04:39.045725 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.045703    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 23:04:39.045799 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.045540    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-6nlxz\""
Sep 17 23:04:39.046050 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.046034    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 23:04:39.055090 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.055069    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"]
Sep 17 23:04:39.077662 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.077636    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9rfn\" (UniqueName: \"kubernetes.io/projected/d58d96d7-b22e-405c-878e-ba149683b0c3-kube-api-access-j9rfn\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p\" (UID: \"d58d96d7-b22e-405c-878e-ba149683b0c3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"
Sep 17 23:04:39.077762 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.077690    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58d96d7-b22e-405c-878e-ba149683b0c3-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p\" (UID: \"d58d96d7-b22e-405c-878e-ba149683b0c3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"
Sep 17 23:04:39.178335 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.178313    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9rfn\" (UniqueName: \"kubernetes.io/projected/d58d96d7-b22e-405c-878e-ba149683b0c3-kube-api-access-j9rfn\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p\" (UID: \"d58d96d7-b22e-405c-878e-ba149683b0c3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"
Sep 17 23:04:39.178449 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.178361    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58d96d7-b22e-405c-878e-ba149683b0c3-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p\" (UID: \"d58d96d7-b22e-405c-878e-ba149683b0c3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"
Sep 17 23:04:39.180960 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.180931    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58d96d7-b22e-405c-878e-ba149683b0c3-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p\" (UID: \"d58d96d7-b22e-405c-878e-ba149683b0c3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"
Sep 17 23:04:39.187065 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.187038    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9rfn\" (UniqueName: \"kubernetes.io/projected/d58d96d7-b22e-405c-878e-ba149683b0c3-kube-api-access-j9rfn\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p\" (UID: \"d58d96d7-b22e-405c-878e-ba149683b0c3\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"
Sep 17 23:04:39.352181 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.352108    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"
Sep 17 23:04:39.640174 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:39.640148    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"]
Sep 17 23:04:39.641842 ip-10-0-142-197 kubenswrapper[2567]: W0917 23:04:39.641812    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58d96d7_b22e_405c_878e_ba149683b0c3.slice/crio-ab619a865626aec295ed0aedb6df473c98f19b24872f97a4f7399fcd054f65c0 WatchSource:0}: Error finding container ab619a865626aec295ed0aedb6df473c98f19b24872f97a4f7399fcd054f65c0: Status 404 returned error can't find the container with id ab619a865626aec295ed0aedb6df473c98f19b24872f97a4f7399fcd054f65c0
Sep 17 23:04:40.480090 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:40.480056    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p" event={"ID":"d58d96d7-b22e-405c-878e-ba149683b0c3","Type":"ContainerStarted","Data":"78130bf0d493604c9cef213c185458957cbbf806d623c619c0fe07a842a96c55"}
Sep 17 23:04:40.480485 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:40.480109    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p" event={"ID":"d58d96d7-b22e-405c-878e-ba149683b0c3","Type":"ContainerStarted","Data":"ab619a865626aec295ed0aedb6df473c98f19b24872f97a4f7399fcd054f65c0"}
Sep 17 23:04:40.480485 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:40.480170    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"
Sep 17 23:04:40.495681 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:40.495641    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p" podStartSLOduration=1.495627497 podStartE2EDuration="1.495627497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:04:40.494487661 +0000 UTC m=+3957.503417198" watchObservedRunningTime="2026-09-17 23:04:40.495627497 +0000 UTC m=+3957.504557038"
Sep 17 23:04:48.225325 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:48.225284    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845" podUID="5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:04:58.225097 ip-10-0-142-197 kubenswrapper[2567]: W0917 23:04:58.225068    2567 logging.go:55] [core] [Channel #1505 SubChannel #1506]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.65:9003", ServerName: "10.133.0.65:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.65:9003: connect: connection refused"
Sep 17 23:04:58.314859 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.314832    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"
Sep 17 23:04:58.413504 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.413410    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzlgh\" (UniqueName: \"kubernetes.io/projected/5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d-kube-api-access-dzlgh\") pod \"5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d\" (UID: \"5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d\") "
Sep 17 23:04:58.413653 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.413507    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d-tls-certs\") pod \"5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d\" (UID: \"5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d\") "
Sep 17 23:04:58.415597 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.415565    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d-kube-api-access-dzlgh" (OuterVolumeSpecName: "kube-api-access-dzlgh") pod "5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d" (UID: "5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d"). InnerVolumeSpecName "kube-api-access-dzlgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:04:58.415597 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.415588    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d" (UID: "5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:04:58.514453 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.514427    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:04:58.514453 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.514450    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzlgh\" (UniqueName: \"kubernetes.io/projected/5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d-kube-api-access-dzlgh\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:04:58.549519 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.549489    2567 generic.go:356] "Generic (PLEG): container finished" podID="5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d" containerID="5d20db50a0e81ef106bb0d94303b4e98040fb53d0b23c30b2392e0a013992e1a" exitCode=0
Sep 17 23:04:58.549624 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.549535    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"
Sep 17 23:04:58.549667 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.549537    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845" event={"ID":"5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d","Type":"ContainerDied","Data":"5d20db50a0e81ef106bb0d94303b4e98040fb53d0b23c30b2392e0a013992e1a"}
Sep 17 23:04:58.549723 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.549696    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845" event={"ID":"5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d","Type":"ContainerDied","Data":"21fd2fcd080f414acf2bda80b224f4706ec815a91445f8ddc15d5f60061cef72"}
Sep 17 23:04:58.549829 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.549730    2567 scope.go:117] "RemoveContainer" containerID="5d20db50a0e81ef106bb0d94303b4e98040fb53d0b23c30b2392e0a013992e1a"
Sep 17 23:04:58.558930 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.558912    2567 scope.go:117] "RemoveContainer" containerID="5d20db50a0e81ef106bb0d94303b4e98040fb53d0b23c30b2392e0a013992e1a"
Sep 17 23:04:58.559216 ip-10-0-142-197 kubenswrapper[2567]: E0917 23:04:58.559192    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d20db50a0e81ef106bb0d94303b4e98040fb53d0b23c30b2392e0a013992e1a\": container with ID starting with 5d20db50a0e81ef106bb0d94303b4e98040fb53d0b23c30b2392e0a013992e1a not found: ID does not exist" containerID="5d20db50a0e81ef106bb0d94303b4e98040fb53d0b23c30b2392e0a013992e1a"
Sep 17 23:04:58.559344 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.559226    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d20db50a0e81ef106bb0d94303b4e98040fb53d0b23c30b2392e0a013992e1a"} err="failed to get container status \"5d20db50a0e81ef106bb0d94303b4e98040fb53d0b23c30b2392e0a013992e1a\": rpc error: code = NotFound desc = could not find container \"5d20db50a0e81ef106bb0d94303b4e98040fb53d0b23c30b2392e0a013992e1a\": container with ID starting with 5d20db50a0e81ef106bb0d94303b4e98040fb53d0b23c30b2392e0a013992e1a not found: ID does not exist"
Sep 17 23:04:58.572064 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.572040    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"]
Sep 17 23:04:58.578034 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:58.577992    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845"]
Sep 17 23:04:59.224164 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:59.224118    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-lm845" podUID="5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.65:9003\" within 1s: context deadline exceeded"
Sep 17 23:04:59.639254 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:04:59.639219    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d" path="/var/lib/kubelet/pods/5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d/volumes"
Sep 17 23:05:11.485356 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:11.485327    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"
Sep 17 23:05:12.870818 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:12.870781    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"]
Sep 17 23:05:21.485512 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:21.485466    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p" podUID="d58d96d7-b22e-405c-878e-ba149683b0c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:05:27.889865 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:27.889810    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p" podUID="d58d96d7-b22e-405c-878e-ba149683b0c3" containerName="main" containerID="cri-o://78130bf0d493604c9cef213c185458957cbbf806d623c619c0fe07a842a96c55" gracePeriod=45
Sep 17 23:05:31.485107 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:31.485047    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p" podUID="d58d96d7-b22e-405c-878e-ba149683b0c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:05:41.485030 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:41.484972    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p" podUID="d58d96d7-b22e-405c-878e-ba149683b0c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:05:41.485428 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:41.485087    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"
Sep 17 23:05:51.484842 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:51.484800    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p" podUID="d58d96d7-b22e-405c-878e-ba149683b0c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 23:05:58.130954 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.130930    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"
Sep 17 23:05:58.225196 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.225128    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9rfn\" (UniqueName: \"kubernetes.io/projected/d58d96d7-b22e-405c-878e-ba149683b0c3-kube-api-access-j9rfn\") pod \"d58d96d7-b22e-405c-878e-ba149683b0c3\" (UID: \"d58d96d7-b22e-405c-878e-ba149683b0c3\") "
Sep 17 23:05:58.225196 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.225186    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58d96d7-b22e-405c-878e-ba149683b0c3-tls-certs\") pod \"d58d96d7-b22e-405c-878e-ba149683b0c3\" (UID: \"d58d96d7-b22e-405c-878e-ba149683b0c3\") "
Sep 17 23:05:58.227372 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.227347    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d58d96d7-b22e-405c-878e-ba149683b0c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d58d96d7-b22e-405c-878e-ba149683b0c3" (UID: "d58d96d7-b22e-405c-878e-ba149683b0c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:05:58.227470 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.227450    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58d96d7-b22e-405c-878e-ba149683b0c3-kube-api-access-j9rfn" (OuterVolumeSpecName: "kube-api-access-j9rfn") pod "d58d96d7-b22e-405c-878e-ba149683b0c3" (UID: "d58d96d7-b22e-405c-878e-ba149683b0c3"). InnerVolumeSpecName "kube-api-access-j9rfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:05:58.325548 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.325514    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58d96d7-b22e-405c-878e-ba149683b0c3-tls-certs\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:05:58.325548 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.325547    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9rfn\" (UniqueName: \"kubernetes.io/projected/d58d96d7-b22e-405c-878e-ba149683b0c3-kube-api-access-j9rfn\") on node \"ip-10-0-142-197.ec2.internal\" DevicePath \"\""
Sep 17 23:05:58.756016 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.755974    2567 generic.go:356] "Generic (PLEG): container finished" podID="d58d96d7-b22e-405c-878e-ba149683b0c3" containerID="78130bf0d493604c9cef213c185458957cbbf806d623c619c0fe07a842a96c55" exitCode=0
Sep 17 23:05:58.756175 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.756038    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"
Sep 17 23:05:58.756175 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.756038    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p" event={"ID":"d58d96d7-b22e-405c-878e-ba149683b0c3","Type":"ContainerDied","Data":"78130bf0d493604c9cef213c185458957cbbf806d623c619c0fe07a842a96c55"}
Sep 17 23:05:58.756175 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.756110    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p" event={"ID":"d58d96d7-b22e-405c-878e-ba149683b0c3","Type":"ContainerDied","Data":"ab619a865626aec295ed0aedb6df473c98f19b24872f97a4f7399fcd054f65c0"}
Sep 17 23:05:58.756175 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.756143    2567 scope.go:117] "RemoveContainer" containerID="78130bf0d493604c9cef213c185458957cbbf806d623c619c0fe07a842a96c55"
Sep 17 23:05:58.765215 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.765192    2567 scope.go:117] "RemoveContainer" containerID="78130bf0d493604c9cef213c185458957cbbf806d623c619c0fe07a842a96c55"
Sep 17 23:05:58.765486 ip-10-0-142-197 kubenswrapper[2567]: E0917 23:05:58.765465    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78130bf0d493604c9cef213c185458957cbbf806d623c619c0fe07a842a96c55\": container with ID starting with 78130bf0d493604c9cef213c185458957cbbf806d623c619c0fe07a842a96c55 not found: ID does not exist" containerID="78130bf0d493604c9cef213c185458957cbbf806d623c619c0fe07a842a96c55"
Sep 17 23:05:58.765530 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.765496    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78130bf0d493604c9cef213c185458957cbbf806d623c619c0fe07a842a96c55"} err="failed to get container status \"78130bf0d493604c9cef213c185458957cbbf806d623c619c0fe07a842a96c55\": rpc error: code = NotFound desc = could not find container \"78130bf0d493604c9cef213c185458957cbbf806d623c619c0fe07a842a96c55\": container with ID starting with 78130bf0d493604c9cef213c185458957cbbf806d623c619c0fe07a842a96c55 not found: ID does not exist"
Sep 17 23:05:58.776586 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.776564    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"]
Sep 17 23:05:58.779643 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:58.779622    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c996s7p"]
Sep 17 23:05:59.638895 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:05:59.638861    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d58d96d7-b22e-405c-878e-ba149683b0c3" path="/var/lib/kubelet/pods/d58d96d7-b22e-405c-878e-ba149683b0c3/volumes"
Sep 17 23:06:31.338664 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:31.338582    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-pkz4h_5329a7fa-75d7-4297-9c19-912d2496ceb2/authorino/0.log"
Sep 17 23:06:31.384027 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:31.383986    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xz29h_62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce/kuadrant-console-plugin/0.log"
Sep 17 23:06:37.484048 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.484008    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r2jh9/must-gather-2bf6p"]
Sep 17 23:06:37.484427 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.484293    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d" containerName="main"
Sep 17 23:06:37.484427 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.484304    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d" containerName="main"
Sep 17 23:06:37.484427 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.484324    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d58d96d7-b22e-405c-878e-ba149683b0c3" containerName="main"
Sep 17 23:06:37.484427 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.484329    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d58d96d7-b22e-405c-878e-ba149683b0c3" containerName="main"
Sep 17 23:06:37.484427 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.484380    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5810dd9a-5cb4-4b8a-9b47-27dd64c9d77d" containerName="main"
Sep 17 23:06:37.484427 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.484388    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d58d96d7-b22e-405c-878e-ba149683b0c3" containerName="main"
Sep 17 23:06:37.487248 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.487229    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r2jh9/must-gather-2bf6p"
Sep 17 23:06:37.489722 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.489697    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r2jh9\"/\"kube-root-ca.crt\""
Sep 17 23:06:37.489722 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.489711    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r2jh9\"/\"openshift-service-ca.crt\""
Sep 17 23:06:37.490654 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.490637    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r2jh9\"/\"default-dockercfg-gmh4m\""
Sep 17 23:06:37.495254 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.495234    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r2jh9/must-gather-2bf6p"]
Sep 17 23:06:37.608899 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.608866    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b5178c14-ff49-4006-8929-73dc194def49-must-gather-output\") pod \"must-gather-2bf6p\" (UID: \"b5178c14-ff49-4006-8929-73dc194def49\") " pod="openshift-must-gather-r2jh9/must-gather-2bf6p"
Sep 17 23:06:37.609067 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.608937    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzck9\" (UniqueName: \"kubernetes.io/projected/b5178c14-ff49-4006-8929-73dc194def49-kube-api-access-xzck9\") pod \"must-gather-2bf6p\" (UID: \"b5178c14-ff49-4006-8929-73dc194def49\") " pod="openshift-must-gather-r2jh9/must-gather-2bf6p"
Sep 17 23:06:37.709441 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.709412    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b5178c14-ff49-4006-8929-73dc194def49-must-gather-output\") pod \"must-gather-2bf6p\" (UID: \"b5178c14-ff49-4006-8929-73dc194def49\") " pod="openshift-must-gather-r2jh9/must-gather-2bf6p"
Sep 17 23:06:37.709593 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.709491    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzck9\" (UniqueName: \"kubernetes.io/projected/b5178c14-ff49-4006-8929-73dc194def49-kube-api-access-xzck9\") pod \"must-gather-2bf6p\" (UID: \"b5178c14-ff49-4006-8929-73dc194def49\") " pod="openshift-must-gather-r2jh9/must-gather-2bf6p"
Sep 17 23:06:37.709833 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.709815    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b5178c14-ff49-4006-8929-73dc194def49-must-gather-output\") pod \"must-gather-2bf6p\" (UID: \"b5178c14-ff49-4006-8929-73dc194def49\") " pod="openshift-must-gather-r2jh9/must-gather-2bf6p"
Sep 17 23:06:37.716685 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.716657    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzck9\" (UniqueName: \"kubernetes.io/projected/b5178c14-ff49-4006-8929-73dc194def49-kube-api-access-xzck9\") pod \"must-gather-2bf6p\" (UID: \"b5178c14-ff49-4006-8929-73dc194def49\") " pod="openshift-must-gather-r2jh9/must-gather-2bf6p"
Sep 17 23:06:37.797403 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:37.797321    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r2jh9/must-gather-2bf6p"
Sep 17 23:06:38.068121 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:38.068090    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r2jh9/must-gather-2bf6p"]
Sep 17 23:06:38.070493 ip-10-0-142-197 kubenswrapper[2567]: W0917 23:06:38.070469    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5178c14_ff49_4006_8929_73dc194def49.slice/crio-4c80a96853696c807dd6c5732757fdb623ce9d8f3f460af9ddf63fed037f7707 WatchSource:0}: Error finding container 4c80a96853696c807dd6c5732757fdb623ce9d8f3f460af9ddf63fed037f7707: Status 404 returned error can't find the container with id 4c80a96853696c807dd6c5732757fdb623ce9d8f3f460af9ddf63fed037f7707
Sep 17 23:06:38.072227 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:38.072209    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:06:38.893788 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:38.893755    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r2jh9/must-gather-2bf6p" event={"ID":"b5178c14-ff49-4006-8929-73dc194def49","Type":"ContainerStarted","Data":"4c80a96853696c807dd6c5732757fdb623ce9d8f3f460af9ddf63fed037f7707"}
Sep 17 23:06:39.900186 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:39.900149    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r2jh9/must-gather-2bf6p" event={"ID":"b5178c14-ff49-4006-8929-73dc194def49","Type":"ContainerStarted","Data":"da02512380ef45c3a13c82712060f7f1e359a69e35bdb33cf24e30365cf9d12b"}
Sep 17 23:06:39.900186 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:39.900191    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r2jh9/must-gather-2bf6p" event={"ID":"b5178c14-ff49-4006-8929-73dc194def49","Type":"ContainerStarted","Data":"627e3979b80af52911f54b6363ec72a18bdba6c2a69e25af0ba1b4a27e2f192c"}
Sep 17 23:06:39.914707 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:39.914658    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r2jh9/must-gather-2bf6p" podStartSLOduration=2.137799794 podStartE2EDuration="2.914641618s" totalImagesPullingTime="776.841824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:39.91380613 +0000 UTC m=+4076.922735674" watchObservedRunningTime="2026-09-17 23:06:39.914641618 +0000 UTC m=+4076.923571149"
Sep 17 23:06:40.271597 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:40.271566    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-58crn_c02d81d0-2aeb-4cc8-a3e4-cc374c6bd673/global-pull-secret-syncer/0.log"
Sep 17 23:06:40.538032 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:40.537933    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qhmn8_e8cae935-fc5f-46f9-ac7c-582f15caaf4c/konnectivity-agent/0.log"
Sep 17 23:06:40.651040 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:40.650991    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-197.ec2.internal_a6a12a94a946441191f85a154c16cabf/haproxy/0.log"
Sep 17 23:06:45.769178 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:45.769028    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-pkz4h_5329a7fa-75d7-4297-9c19-912d2496ceb2/authorino/0.log"
Sep 17 23:06:45.933959 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:45.933925    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xz29h_62ffd6d4-5733-4e8a-bba5-4ac1364ee3ce/kuadrant-console-plugin/0.log"
Sep 17 23:06:47.761046 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:47.761017    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4zf8r_d5610702-f4cb-4f8d-8a66-fe4d4b77acb8/node-exporter/0.log"
Sep 17 23:06:47.782357 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:47.782326    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4zf8r_d5610702-f4cb-4f8d-8a66-fe4d4b77acb8/kube-rbac-proxy/0.log"
Sep 17 23:06:47.804773 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:47.804743    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4zf8r_d5610702-f4cb-4f8d-8a66-fe4d4b77acb8/init-textfile/0.log"
Sep 17 23:06:49.926896 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:49.926861    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"]
Sep 17 23:06:49.931258 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:49.931194    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:49.937242 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:49.937178    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"]
Sep 17 23:06:50.011674 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.011629    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7968k\" (UniqueName: \"kubernetes.io/projected/0eccf090-30f0-4623-985b-31146d139722-kube-api-access-7968k\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.011884 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.011689    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0eccf090-30f0-4623-985b-31146d139722-proc\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.011884 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.011736    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0eccf090-30f0-4623-985b-31146d139722-podres\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.011884 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.011769    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0eccf090-30f0-4623-985b-31146d139722-lib-modules\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.011884 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.011839    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eccf090-30f0-4623-985b-31146d139722-sys\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.112397 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.112362    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0eccf090-30f0-4623-985b-31146d139722-podres\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.112397 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.112402    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0eccf090-30f0-4623-985b-31146d139722-lib-modules\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.112666 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.112443    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eccf090-30f0-4623-985b-31146d139722-sys\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.112666 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.112470    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7968k\" (UniqueName: \"kubernetes.io/projected/0eccf090-30f0-4623-985b-31146d139722-kube-api-access-7968k\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.112666 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.112487    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0eccf090-30f0-4623-985b-31146d139722-proc\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.112666 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.112526    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0eccf090-30f0-4623-985b-31146d139722-podres\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.112666 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.112549    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eccf090-30f0-4623-985b-31146d139722-sys\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.112666 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.112555    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0eccf090-30f0-4623-985b-31146d139722-proc\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.112666 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.112598    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0eccf090-30f0-4623-985b-31146d139722-lib-modules\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.121497 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.121468    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7968k\" (UniqueName: \"kubernetes.io/projected/0eccf090-30f0-4623-985b-31146d139722-kube-api-access-7968k\") pod \"perf-node-gather-daemonset-cwvq4\" (UID: \"0eccf090-30f0-4623-985b-31146d139722\") " pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.245741 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.245702    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:50.530951 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.527544    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"]
Sep 17 23:06:50.531398 ip-10-0-142-197 kubenswrapper[2567]: W0917 23:06:50.531364    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0eccf090_30f0_4623_985b_31146d139722.slice/crio-104a4aee968518d453f6f627fa8e9e09d45d97fe1c0ce3443ed9f1afbbcd565a WatchSource:0}: Error finding container 104a4aee968518d453f6f627fa8e9e09d45d97fe1c0ce3443ed9f1afbbcd565a: Status 404 returned error can't find the container with id 104a4aee968518d453f6f627fa8e9e09d45d97fe1c0ce3443ed9f1afbbcd565a
Sep 17 23:06:50.982303 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.982244    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4" event={"ID":"0eccf090-30f0-4623-985b-31146d139722","Type":"ContainerStarted","Data":"e1868421fe7af41510c49d9cc96d60793e2e515e9cc499b75d3917aab5d241a3"}
Sep 17 23:06:50.982798 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.982350    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4" event={"ID":"0eccf090-30f0-4623-985b-31146d139722","Type":"ContainerStarted","Data":"104a4aee968518d453f6f627fa8e9e09d45d97fe1c0ce3443ed9f1afbbcd565a"}
Sep 17 23:06:50.983278 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:50.983255    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:06:51.000322 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:51.000262    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4" podStartSLOduration=2.000241859 podStartE2EDuration="2.000241859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:06:50.998556814 +0000 UTC m=+4088.007486354" watchObservedRunningTime="2026-09-17 23:06:51.000241859 +0000 UTC m=+4088.009171402"
Sep 17 23:06:52.103243 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:52.103195    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n7wmb_5d9abd2c-c138-4356-963c-160dfdb23d5a/dns/0.log"
Sep 17 23:06:52.124272 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:52.124244    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n7wmb_5d9abd2c-c138-4356-963c-160dfdb23d5a/kube-rbac-proxy/0.log"
Sep 17 23:06:52.176923 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:52.176890    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n4zhm_149b4a0e-4466-46a6-9e51-d2450702c4d0/dns-node-resolver/0.log"
Sep 17 23:06:52.740198 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:52.740164    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6fdccdf588-rpvrh_7a2fb062-67ac-4fc4-85dc-a2ac4accf796/registry/0.log"
Sep 17 23:06:52.813195 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:52.813161    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xc5tg_5dd82d87-0321-4c10-a183-6d6122eda0cc/node-ca/0.log"
Sep 17 23:06:54.300413 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:54.300384    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4d58n_e48a9d5b-0ccd-4f73-8f2b-6e467fcf70c7/serve-healthcheck-canary/0.log"
Sep 17 23:06:54.819454 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:54.819425    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c9h8t_0374325e-9608-4e5b-9551-0f226154fa74/kube-rbac-proxy/0.log"
Sep 17 23:06:54.840053 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:54.840023    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c9h8t_0374325e-9608-4e5b-9551-0f226154fa74/exporter/0.log"
Sep 17 23:06:54.861502 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:54.861471    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c9h8t_0374325e-9608-4e5b-9551-0f226154fa74/extractor/0.log"
Sep 17 23:06:57.890494 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:57.890460    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-65d86c4f4c-v6q7m_0b14d03e-83a1-41d5-a8ff-c8d22a32b48b/manager/0.log"
Sep 17 23:06:58.002359 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:06:58.002331    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r2jh9/perf-node-gather-daemonset-cwvq4"
Sep 17 23:07:04.295424 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:04.295391    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-wcht2_9805e819-611c-4b5d-af80-a380d9004387/migrator/0.log"
Sep 17 23:07:04.315116 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:04.315090    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-wcht2_9805e819-611c-4b5d-af80-a380d9004387/graceful-termination/0.log"
Sep 17 23:07:05.706905 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:05.706871    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9nmn_d856ebe3-2d7c-4833-9576-504511abb585/kube-multus-additional-cni-plugins/0.log"
Sep 17 23:07:05.726551 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:05.726530    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9nmn_d856ebe3-2d7c-4833-9576-504511abb585/egress-router-binary-copy/0.log"
Sep 17 23:07:05.750029 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:05.749987    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9nmn_d856ebe3-2d7c-4833-9576-504511abb585/cni-plugins/0.log"
Sep 17 23:07:05.774323 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:05.774288    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9nmn_d856ebe3-2d7c-4833-9576-504511abb585/bond-cni-plugin/0.log"
Sep 17 23:07:05.797921 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:05.797901    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9nmn_d856ebe3-2d7c-4833-9576-504511abb585/routeoverride-cni/0.log"
Sep 17 23:07:05.818794 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:05.818778    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9nmn_d856ebe3-2d7c-4833-9576-504511abb585/whereabouts-cni-bincopy/0.log"
Sep 17 23:07:05.842689 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:05.842671    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q9nmn_d856ebe3-2d7c-4833-9576-504511abb585/whereabouts-cni/0.log"
Sep 17 23:07:06.087261 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:06.087223    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-q79d2_bd4fee5c-7168-471c-b29a-f92b77f59b67/kube-multus/0.log"
Sep 17 23:07:06.136291 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:06.136268    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fdxms_94979e3c-1ac8-4800-a60f-3d3552644bcc/network-metrics-daemon/0.log"
Sep 17 23:07:06.154591 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:06.154570    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fdxms_94979e3c-1ac8-4800-a60f-3d3552644bcc/kube-rbac-proxy/0.log"
Sep 17 23:07:07.013452 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:07.013420    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kp5t_50ab8fb7-7f5b-401f-a16b-740d063b3ee6/ovn-controller/0.log"
Sep 17 23:07:07.029757 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:07.029736    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kp5t_50ab8fb7-7f5b-401f-a16b-740d063b3ee6/ovn-acl-logging/0.log"
Sep 17 23:07:07.051793 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:07.051772    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kp5t_50ab8fb7-7f5b-401f-a16b-740d063b3ee6/kube-rbac-proxy-node/0.log"
Sep 17 23:07:07.074611 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:07.074582    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kp5t_50ab8fb7-7f5b-401f-a16b-740d063b3ee6/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 23:07:07.096542 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:07.096513    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kp5t_50ab8fb7-7f5b-401f-a16b-740d063b3ee6/northd/0.log"
Sep 17 23:07:07.120519 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:07.120489    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kp5t_50ab8fb7-7f5b-401f-a16b-740d063b3ee6/nbdb/0.log"
Sep 17 23:07:07.143109 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:07.143080    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kp5t_50ab8fb7-7f5b-401f-a16b-740d063b3ee6/sbdb/0.log"
Sep 17 23:07:07.212989 ip-10-0-142-197 kubenswrapper[2567]: I0917 23:07:07.212962    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2kp5t_50ab8fb7-7f5b-401f-a16b-740d063b3ee6/ovnkube-controller/0.log"