Sep 16 18:44:33.512976 ip-10-0-135-244 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:44:33.861205 ip-10-0-135-244 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:33.861205 ip-10-0-135-244 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:44:33.861205 ip-10-0-135-244 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:33.861205 ip-10-0-135-244 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:44:33.861205 ip-10-0-135-244 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:33.862929 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.862843    2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:44:33.866374 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866359    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:33.866374 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866374    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866378    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866382    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866385    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866388    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866391    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866393    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866396    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866399    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866402    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866404    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866407    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866410    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866412    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866415    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866417    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866420    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866423    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866426    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866434    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:33.866455 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866437    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866440    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866442    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866445    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866447    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866450    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866453    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866455    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866459    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866461    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866466    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866470    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866474    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866477    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866479    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866482    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866485    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866488    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866490    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:33.866950 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866493    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866496    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866499    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866501    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866503    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866506    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866508    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866511    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866513    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866516    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866518    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866521    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866524    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866526    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866529    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866532    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866535    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866537    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866540    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:33.867427 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866543    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866545    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866548    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866551    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866555    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866559    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866562    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866564    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866567    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866569    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866572    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866574    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866577    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866579    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866582    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866585    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866587    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866590    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866592    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:33.867895 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866595    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866600    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866603    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866606    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866609    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866613    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866616    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866619    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866622    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866626    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866628    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866631    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866634    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866636    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866639    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866642    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866644    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866647    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866649    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866652    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:33.868349 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.866654    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867475    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867482    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867485    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867488    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867491    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867494    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867497    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867500    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867508    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867513    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867517    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867519    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867522    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867526    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867531    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867534    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867537    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867540    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:33.868877 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867543    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867546    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867549    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867551    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867554    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867556    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867559    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867562    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867565    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867567    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867569    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867572    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867575    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867577    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867580    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867583    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867585    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867588    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867590    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867593    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:33.869373 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867595    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867598    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867601    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867610    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867613    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867616    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867620    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867623    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867626    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867628    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867631    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867634    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867636    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867640    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867643    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867646    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867649    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867652    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867654    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:33.869874 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867657    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867660    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867662    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867665    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867667    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867670    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867674    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867676    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867678    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867681    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867699    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867702    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867705    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867708    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867710    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867713    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867715    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867724    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867727    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867729    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:33.870344 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867732    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867734    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867737    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867740    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867742    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867745    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867748    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867751    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867754    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867756    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867759    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867762    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867764    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867767    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867770    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867772    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867775    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867777    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867780    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867782    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:33.870853 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867785    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.867787    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868147    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868151    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868154    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868157    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868159    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868162    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868165    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868168    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868170    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868173    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868176    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868178    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868181    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868186    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868188    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868192    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868196    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868200    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:33.871328 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868202    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868205    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868208    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868211    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868213    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868216    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868219    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868221    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868223    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868226    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868229    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868231    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868234    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868236    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868239    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868241    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868244    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868246    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868249    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868254    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:33.871828 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868256    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868259    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868262    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868264    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868267    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868269    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868272    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868277    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868280    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868282    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868285    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868287    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868290    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868292    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868295    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868298    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868300    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868302    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868305    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868307    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:33.872315 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868310    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868313    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868315    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868318    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868320    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868323    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868325    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868328    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868331    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868333    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868336    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868340    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868342    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868345    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868348    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868350    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868353    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868356    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868358    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:33.872804 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868362    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868367    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868370    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868373    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868376    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868379    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868382    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868384    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868387    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868389    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868392    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868394    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868397    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868400    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868402    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868405    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868408    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868411    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868413    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:33.873267 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868416    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868418    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.868421    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868495    2570 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868527    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868533    2570 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868537    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868541    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868544    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868548    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868553    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868556    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868559    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868565    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868568    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868571    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868574    2570 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868577    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868580    2570 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868582    2570 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868585    2570 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868589    2570 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868592    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868595    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:44:33.873740 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868598    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868601    2570 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868605    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868608    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868611    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868614    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868617    2570 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868620    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868623    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868626    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868629    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868635    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868638    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868641    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868644    2570 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868647    2570 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868650    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868653    2570 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868656    2570 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868660    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868663    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868667    2570 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868671    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868674    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868677    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:44:33.874384 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868680    2570 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868695    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868700    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868703    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868706    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868709    2570 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868712    2570 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868715    2570 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868719    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868722    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868725    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868728    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868731    2570 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868734    2570 flags.go:64] FLAG: --help="false"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868737    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-135-244.ec2.internal"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868740    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868743    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868746    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868749    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868752    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868756    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868759    2570 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868762    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:44:33.875057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868765    2570 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868768    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868771    2570 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868774    2570 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868777    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868781    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868784    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868787    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868790    2570 flags.go:64] FLAG: --lock-file=""
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868793    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868795    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868798    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868803    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868806    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868809    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868812    2570 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868815    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868818    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868821    2570 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868824    2570 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868828    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868831    2570 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868835    2570 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868837    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868840    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:44:33.875656 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868843    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868846    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868849    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868852    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868855    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868862    2570 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868865    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868868    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868871    2570 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868874    2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868879    2570 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868882    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868886    2570 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868890    2570 flags.go:64] FLAG: --port="10250"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868893    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868896    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a31c9e52efb5605f"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868899    2570 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868902    2570 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868905    2570 flags.go:64] FLAG: --register-node="true"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868908    2570 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868911    2570 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868914    2570 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868917    2570 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868920    2570 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868923    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:44:33.876404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868926    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868929    2570 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868932    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868935    2570 flags.go:64] FLAG: --runonce="false"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868938    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868941    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868943    2570 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868946    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868949    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868952    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868955    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868959    2570 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868962    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868965    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868968    2570 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868971    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868974    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868977    2570 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868979    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868985    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868989    2570 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868991    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868995    2570 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.868998    2570 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.869001    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:44:33.877366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.869004    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.869006    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.869009    2570 flags.go:64] FLAG: --v="2"
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.869013    2570 flags.go:64] FLAG: --version="false"
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.869016    2570 flags.go:64] FLAG: --vmodule=""
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.869020    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.869023    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869132    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869136    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869140    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869143    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869146    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869149    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869151    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869154    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869156    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869159    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869161    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869165    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869168    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869170    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869173    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:33.878019 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869176    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869178    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869181    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869183    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869186    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869189    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869192    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869195    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869198    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869200    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869203    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869206    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869209    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869211    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869214    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869216    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869231    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869236    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869239    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869242    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:33.878546 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869245    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869248    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869251    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869254    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869257    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869260    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869263    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869265    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869269    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869272    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869275    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869278    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869280    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869283    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869286    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869288    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869291    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869294    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869296    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869300    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:33.879083 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869303    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869305    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869308    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869310    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869313    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869315    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869318    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869321    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869323    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869326    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869330    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869333    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869336    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869339    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869343    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869346    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869349    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869352    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869354    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:33.879572 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869357    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869361    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869364    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869367    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869369    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869372    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869375    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869378    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869380    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869383    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869386    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869388    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869391    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869394    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869397    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869399    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869402    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869404    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869407    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869411    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:33.880118 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869414    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869417    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869420    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869422    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.869425    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.869919    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.878641    2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.878656    2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.879322    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.879331    2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879402    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879407    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879410    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879413    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879416    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:33.880611 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879419    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879423    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879427    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879431    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879434    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879437    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879439    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879442    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879445    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879447    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879450    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879453    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879456    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879458    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879461    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879463    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879466    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879468    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879471    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:33.880993 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879473    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879476    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879479    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879482    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879484    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879488    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879491    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879494    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879496    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879499    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879502    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879504    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879506    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879509    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879511    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879514    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879517    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879520    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879522    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:33.881439 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879525    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879528    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879530    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879533    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879535    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879539    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879543    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879545    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879548    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879550    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879553    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879556    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879558    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879561    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879564    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879566    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879569    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879571    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879575    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879578    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:33.881927 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879581    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879584    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879586    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879589    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879591    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879594    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879597    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879599    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879602    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879604    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879607    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879609    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879611    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879614    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879616    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879619    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879623    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879625    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879628    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879631    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:33.882447 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879633    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879636    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879638    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879641    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879643    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879647    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879650    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879653    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879656    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879659    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879662    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879665    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879667    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879670    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879673    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:33.882949 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879676    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.879681    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879805    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879810    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879813    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879816    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879819    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879822    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879824    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879827    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879829    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879832    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879835    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879837    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879840    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:33.883301 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879843    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879846    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879848    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879851    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879853    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879856    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879858    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879861    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879865    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879868    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879871    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879874    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879877    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879880    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879883    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879886    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879888    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879891    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879894    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879896    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:33.883674 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879899    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879901    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879904    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879906    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879909    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879911    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879914    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879916    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879918    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879921    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879923    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879926    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879929    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879932    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879934    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879937    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879939    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879942    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879944    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:33.884165 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879946    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879949    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879953    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879956    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879959    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879961    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879964    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879967    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879969    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879972    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879974    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879977    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879979    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879982    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879984    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879987    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879989    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879992    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879994    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:33.884612 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.879997    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880000    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880002    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880004    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880007    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880009    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880012    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880015    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880018    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880020    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880023    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880026    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880029    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880033    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880035    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880038    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880040    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880043    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880045    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880048    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:33.885084 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880051    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:33.885564 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880053    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:33.885564 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880056    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:33.885564 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880058    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:33.885564 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880061    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:33.885564 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880063    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:33.885564 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880066    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:33.885564 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:33.880068    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:33.885564 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.880073    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:33.885564 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.880189    2570 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:44:33.885564 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.883725    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:44:33.885564 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.884506    2570 server.go:1019] "Starting client certificate rotation"
Sep 16 18:44:33.885564 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.884598    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:44:33.885564 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.884650    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:44:33.886862 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.886849    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:44:33.890720 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.890708    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:44:33.901086 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.901070    2570 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:44:33.905350 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.905333    2570 log.go:25] "Validated CRI v1 image API"
Sep 16 18:44:33.906791 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.906739    2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:44:33.909382 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.909362    2570 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a2c2716a-62ce-49f5-ab07-6f1a78239700:/dev/nvme0n1p3 d8410e18-c7d9-464a-aca9-0618faad4390:/dev/nvme0n1p4]
Sep 16 18:44:33.909453 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.909380    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:44:33.914090 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.913760    2570 manager.go:216] Machine: {Timestamp:2026-09-16 18:44:33.913071305 +0000 UTC m=+0.297630518 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101993 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2128d05ce71f0eee0b83484eb86be6 SystemUUID:ec2128d0-5ce7-1f0e-ee0b-83484eb86be6 BootID:cde138c5-89bb-413d-a89c-62a88bdc318c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ce:88:11:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ce:88:11:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:cf:0c:de:f9:a9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:44:33.914090 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.914084    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:44:33.914221 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.914161    2570 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:44:33.915663 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.915640    2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:44:33.915819 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.915665    2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-244.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:44:33.916311 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.916302    2570 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:44:33.916340 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.916314    2570 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:44:33.916340 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.916321    2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:44:33.916903 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.916893    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:44:33.917558 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.917549    2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:44:33.920065 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.920053    2570 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:44:33.920168 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.920159    2570 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:44:33.920208 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.920189    2570 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:44:33.920208 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.920201    2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:44:33.920260 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.920223    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:44:33.920260 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.920236    2570 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:44:33.920260 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.920247    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:44:33.920629 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.920617    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:44:33.920696 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.920678    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:44:33.920960 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.920941    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:44:33.922447 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.922431    2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:44:33.923928 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.923915    2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:44:33.923977 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.923944    2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:44:33.925194 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.925182    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:44:33.925248 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.925198    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:44:33.925248 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.925204    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:44:33.925248 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.925210    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:44:33.925248 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.925219    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:44:33.925248 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.925225    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:44:33.925248 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.925238    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:44:33.925248 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.925243    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:44:33.925248 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.925250    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:44:33.925444 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.925256    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:44:33.925444 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.925264    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:44:33.925444 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.925269    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:44:33.925788 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.925776    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:44:33.925841 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.925792    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:44:33.929377 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.929356    2570 server.go:1268] "Started kubelet"
Sep 16 18:44:33.929455 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.929426    2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:44:33.929617 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.929559    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:44:33.929743 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.929666    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:44:33.929810 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.929767    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-244.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:44:33.929810 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:33.929765    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:44:33.929911 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:33.929881    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:44:33.930309 ip-10-0-135-244 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:44:33.931313 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.931299    2570 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:44:33.931943 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.931931    2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:44:33.935251 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.935234    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:44:33.935339 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:33.935274    2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:44:33.935422 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:33.934584    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-244.ec2.internal.18d5e16a5f2ec441  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-244.ec2.internal,UID:ip-10-0-135-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-244.ec2.internal,},FirstTimestamp:2026-09-16 18:44:33.929323585 +0000 UTC m=+0.313882800,LastTimestamp:2026-09-16 18:44:33.929323585 +0000 UTC m=+0.313882800,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-244.ec2.internal,}"
Sep 16 18:44:33.935703 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.935676    2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:44:33.936002 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:33.935981    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 16 18:44:33.936421 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.936390    2570 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:44:33.936421 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.936414    2570 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:44:33.936545 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.936466    2570 factory.go:55] Registering systemd factory
Sep 16 18:44:33.936545 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.936386    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:44:33.936545 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.936480    2570 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:44:33.936545 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.936533    2570 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:44:33.936545 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.936541    2570 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:44:33.936769 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.936720    2570 factory.go:153] Registering CRI-O factory
Sep 16 18:44:33.936769 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.936734    2570 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:44:33.936873 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.936789    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:44:33.936873 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.936808    2570 factory.go:103] Registering Raw factory
Sep 16 18:44:33.936873 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.936823    2570 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:44:33.937191 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.937176    2570 manager.go:349] Starting recovery of all containers
Sep 16 18:44:33.943509 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:33.942645    2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:44:33.943580 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:33.942676    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:44:33.949213 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.949193    2570 manager.go:354] Recovery completed
Sep 16 18:44:33.953890 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.953877    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:33.955555 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.955536    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:33.955647 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.955563    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:33.955647 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.955576    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:33.956204 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.956186    2570 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:44:33.956204 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.956205    2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:44:33.956305 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.956222    2570 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:44:33.956305 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.956188    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:33.956886 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.956873    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:33.956940 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.956897    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:33.956940 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.956909    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:33.958013 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:33.957957    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-244.ec2.internal.18d5e16a60bef57a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-244.ec2.internal,UID:ip-10-0-135-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-244.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-244.ec2.internal,},FirstTimestamp:2026-09-16 18:44:33.955550586 +0000 UTC m=+0.340109800,LastTimestamp:2026-09-16 18:44:33.955550586 +0000 UTC m=+0.340109800,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-244.ec2.internal,}"
Sep 16 18:44:33.966937 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:33.966855    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-244.ec2.internal.18d5e16a60bf4300  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-244.ec2.internal,UID:ip-10-0-135-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-244.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-244.ec2.internal,},FirstTimestamp:2026-09-16 18:44:33.955570432 +0000 UTC m=+0.340129649,LastTimestamp:2026-09-16 18:44:33.955570432 +0000 UTC m=+0.340129649,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-244.ec2.internal,}"
Sep 16 18:44:33.973016 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.973000    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cg8tw"
Sep 16 18:44:33.977005 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:33.976946    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-244.ec2.internal.18d5e16a60bf6a2b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-244.ec2.internal,UID:ip-10-0-135-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-244.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-244.ec2.internal,},FirstTimestamp:2026-09-16 18:44:33.955580459 +0000 UTC m=+0.340139673,LastTimestamp:2026-09-16 18:44:33.955580459 +0000 UTC m=+0.340139673,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-244.ec2.internal,}"
Sep 16 18:44:33.981600 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:33.981585    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cg8tw"
Sep 16 18:44:34.036915 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.036894    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 16 18:44:34.102621 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.102608    2570 policy_none.go:49] "None policy: Start"
Sep 16 18:44:34.102706 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.102629    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:44:34.102706 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.102640    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:44:34.103621 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.103611    2570 policy_none.go:47] "Start"
Sep 16 18:44:34.136985 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.136969    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 16 18:44:34.141249 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.141237    2570 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:44:34.141321 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.141266    2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:44:34.141321 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.141274    2570 server.go:87] "Starting device plugin registration server"
Sep 16 18:44:34.141553 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.141536    2570 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:44:34.141635 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.141548    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:44:34.141705 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.141644    2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:44:34.141763 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.141730    2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:44:34.141763 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.141737    2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:44:34.142280 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.142253    2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:44:34.142373 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.142295    2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 16 18:44:34.149971 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.149952    2570 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-244.ec2.internal\" not found" node="ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.154039 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.154018    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:44:34.155127 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.155114    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:44:34.155182 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.155157    2570 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:44:34.155182 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.155178    2570 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:44:34.155263 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.155213    2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:44:34.158369 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.158354    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:34.242420 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.242396    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:34.243151 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.243137    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:34.243213 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.243161    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:34.243213 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.243171    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:34.243213 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.243196    2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.250368 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.250356    2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.250419 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.250373    2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-244.ec2.internal\": node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 16 18:44:34.255390 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.255373    2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal"]
Sep 16 18:44:34.255464 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.255442    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:34.256649 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.256635    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:34.256730 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.256659    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:34.256730 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.256671    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:34.257757 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.257746    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:34.257867 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.257854    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.257906 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.257883    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:34.258415 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.258401    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:34.258415 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.258419    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:34.258543 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.258430    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:34.258543 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.258459    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:34.258543 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.258477    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:34.258543 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.258491    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:34.259478 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.259464    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.259527 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.259489    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:34.260199 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.260184    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:34.260265 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.260203    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:34.260265 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.260211    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:34.274193 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.274177    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-244.ec2.internal\" not found" node="ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.277540 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.277524    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 16 18:44:34.278445 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.278430    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-244.ec2.internal\" not found" node="ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.338037 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.338017    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8fd94c484fe2786e35424aca732ed6cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal\" (UID: \"8fd94c484fe2786e35424aca732ed6cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.338113 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.338040    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fd94c484fe2786e35424aca732ed6cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal\" (UID: \"8fd94c484fe2786e35424aca732ed6cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.338113 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.338057    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f223e05d53cb9f1d1ecb96b3f640ab21-config\") pod \"kube-apiserver-proxy-ip-10-0-135-244.ec2.internal\" (UID: \"f223e05d53cb9f1d1ecb96b3f640ab21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.377860 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.377818    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 16 18:44:34.438495 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.438468    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8fd94c484fe2786e35424aca732ed6cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal\" (UID: \"8fd94c484fe2786e35424aca732ed6cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.438570 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.438501    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fd94c484fe2786e35424aca732ed6cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal\" (UID: \"8fd94c484fe2786e35424aca732ed6cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.438570 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.438526    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f223e05d53cb9f1d1ecb96b3f640ab21-config\") pod \"kube-apiserver-proxy-ip-10-0-135-244.ec2.internal\" (UID: \"f223e05d53cb9f1d1ecb96b3f640ab21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.438570 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.438541    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8fd94c484fe2786e35424aca732ed6cf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal\" (UID: \"8fd94c484fe2786e35424aca732ed6cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.438570 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.438550    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fd94c484fe2786e35424aca732ed6cf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal\" (UID: \"8fd94c484fe2786e35424aca732ed6cf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.438570 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.438555    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f223e05d53cb9f1d1ecb96b3f640ab21-config\") pod \"kube-apiserver-proxy-ip-10-0-135-244.ec2.internal\" (UID: \"f223e05d53cb9f1d1ecb96b3f640ab21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.478603 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.478571    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 16 18:44:34.576037 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.576018    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.579716 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.579676    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 16 18:44:34.580944 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.580927    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:34.680102 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.680048    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 16 18:44:34.771065 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.771045    2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:34.780966 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.780946    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 16 18:44:34.881398 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.881378    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 16 18:44:34.884528 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.884514    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:44:34.884633 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.884617    2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:44:34.884677 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.884658    2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:44:34.981905 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:34.981880    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 16 18:44:34.985304 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.985281    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:39:34 +0000 UTC" deadline="2028-07-03 14:33:58.722042266 +0000 UTC"
Sep 16 18:44:34.985348 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:34.985304    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15739h49m23.736740853s"
Sep 16 18:44:35.082533 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:35.082508    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-244.ec2.internal\" not found"
Sep 16 18:44:35.112992 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.112972    2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:35.133691 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:35.133658    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd94c484fe2786e35424aca732ed6cf.slice/crio-0437b4d8b7a7d08d06b8e67b6dfd62713268912cf5f730912d3d330b69bffd3f WatchSource:0}: Error finding container 0437b4d8b7a7d08d06b8e67b6dfd62713268912cf5f730912d3d330b69bffd3f: Status 404 returned error can't find the container with id 0437b4d8b7a7d08d06b8e67b6dfd62713268912cf5f730912d3d330b69bffd3f
Sep 16 18:44:35.133968 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:35.133936    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf223e05d53cb9f1d1ecb96b3f640ab21.slice/crio-0aee50e445ed810df604b7a2bfa3fcaaa0fcb68a8be47954b7a35f8788b7ebb8 WatchSource:0}: Error finding container 0aee50e445ed810df604b7a2bfa3fcaaa0fcb68a8be47954b7a35f8788b7ebb8: Status 404 returned error can't find the container with id 0aee50e445ed810df604b7a2bfa3fcaaa0fcb68a8be47954b7a35f8788b7ebb8
Sep 16 18:44:35.136053 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.136032    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:35.138288 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.138274    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:44:35.150064 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.150045    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:44:35.150966 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.150951    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal"
Sep 16 18:44:35.158480 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.158439    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal" event={"ID":"8fd94c484fe2786e35424aca732ed6cf","Type":"ContainerStarted","Data":"0437b4d8b7a7d08d06b8e67b6dfd62713268912cf5f730912d3d330b69bffd3f"}
Sep 16 18:44:35.158961 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.158941    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:44:35.159629 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.159607    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal" event={"ID":"f223e05d53cb9f1d1ecb96b3f640ab21","Type":"ContainerStarted","Data":"0aee50e445ed810df604b7a2bfa3fcaaa0fcb68a8be47954b7a35f8788b7ebb8"}
Sep 16 18:44:35.217119 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.217075    2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:35.884521 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.884486    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:35.921153 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.921126    2570 apiserver.go:52] "Watching apiserver"
Sep 16 18:44:35.929779 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.929585    2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:44:35.930097 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.930070    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-2rbsh","kube-system/global-pull-secret-syncer-49tcg","kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75","openshift-dns/node-resolver-jsj4b","openshift-image-registry/node-ca-mzfr8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal","openshift-multus/multus-56tqs","openshift-network-operator/iptables-alerter-n4vdz","kube-system/konnectivity-agent-mrbh7","openshift-cluster-node-tuning-operator/tuned-hnwrq","openshift-multus/multus-additional-cni-plugins-pbd6r","openshift-multus/network-metrics-daemon-bwr7v","openshift-network-diagnostics/network-check-target-47c96"]
Sep 16 18:44:35.931757 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.931728    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.933279 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.933136    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:35.933279 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:35.933207    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:44:35.934530 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.934505    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:35.935502 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.935482    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:44:35.935994 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.935837    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jsj4b"
Sep 16 18:44:35.937500 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.937471    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mzfr8"
Sep 16 18:44:35.938637 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.938507    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-56tqs"
Sep 16 18:44:35.940332 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.940312    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:44:35.941134 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.941078    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:44:35.941977 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.941388    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:44:35.941977 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.941569    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:44:35.941977 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.941735    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:44:35.942610 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.942045    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n4vdz"
Sep 16 18:44:35.944565 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.943180    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:44:35.944565 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.944300    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mrbh7"
Sep 16 18:44:35.945462 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.945438    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:44:35.946574 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.946551    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:35.946955 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.946921    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-run-systemd\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.947067 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.946967    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:35.947067 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947004    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-sys-fs\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:35.947067 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947036    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41b9e54d-41ed-4362-a99a-9998cba62a60-hosts-file\") pod \"node-resolver-jsj4b\" (UID: \"41b9e54d-41ed-4362-a99a-9998cba62a60\") " pod="openshift-dns/node-resolver-jsj4b"
Sep 16 18:44:35.947229 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947071    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-log-socket\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.947229 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947101    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lckmp\" (UniqueName: \"kubernetes.io/projected/41b9e54d-41ed-4362-a99a-9998cba62a60-kube-api-access-lckmp\") pod \"node-resolver-jsj4b\" (UID: \"41b9e54d-41ed-4362-a99a-9998cba62a60\") " pod="openshift-dns/node-resolver-jsj4b"
Sep 16 18:44:35.947229 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947126    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-cni-netd\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.947229 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947166    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-ovn-node-metrics-cert\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.947416 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947245    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-device-dir\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:35.947416 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947284    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b9e54d-41ed-4362-a99a-9998cba62a60-tmp-dir\") pod \"node-resolver-jsj4b\" (UID: \"41b9e54d-41ed-4362-a99a-9998cba62a60\") " pod="openshift-dns/node-resolver-jsj4b"
Sep 16 18:44:35.947416 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947323    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-systemd-units\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.947416 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947359    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-node-log\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.947569 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947470    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-ovnkube-script-lib\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.947569 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947532    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgtvx\" (UniqueName: \"kubernetes.io/projected/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-kube-api-access-kgtvx\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.947766 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947719    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-registration-dir\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:35.947826 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947806    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:35.947864 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947855    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-kubelet\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.947913 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947891    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-run-openvswitch\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.947977 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.947957    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.948046 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948009    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2e821ae-3c87-4fa1-8065-4004d791b3fc-kubelet-config\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:35.948046 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948038    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2e821ae-3c87-4fa1-8065-4004d791b3fc-dbus\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:35.948161 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948082    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xn6j\" (UniqueName: \"kubernetes.io/projected/41f91dbd-52af-482f-88fe-b09f0e5c48cc-kube-api-access-6xn6j\") pod \"node-ca-mzfr8\" (UID: \"41f91dbd-52af-482f-88fe-b09f0e5c48cc\") " pod="openshift-image-registry/node-ca-mzfr8"
Sep 16 18:44:35.948161 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948122    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-run-netns\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.948161 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948125    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:35.948161 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948155    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-run-ovn\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.948418 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948187    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-cni-bin\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.948418 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948242    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-ovnkube-config\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.948418 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948281    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-socket-dir\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:35.948418 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948312    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41f91dbd-52af-482f-88fe-b09f0e5c48cc-host\") pod \"node-ca-mzfr8\" (UID: \"41f91dbd-52af-482f-88fe-b09f0e5c48cc\") " pod="openshift-image-registry/node-ca-mzfr8"
Sep 16 18:44:35.948418 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948344    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-slash\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.948418 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948403    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-var-lib-openvswitch\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.948704 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948432    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-etc-openvswitch\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.948704 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948480    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.948704 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948515    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:35.948892 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948710    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41f91dbd-52af-482f-88fe-b09f0e5c48cc-serviceca\") pod \"node-ca-mzfr8\" (UID: \"41f91dbd-52af-482f-88fe-b09f0e5c48cc\") " pod="openshift-image-registry/node-ca-mzfr8"
Sep 16 18:44:35.948892 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948630    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:44:35.948892 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948795    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-env-overrides\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:35.948994 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.948954    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t92sb\" (UniqueName: \"kubernetes.io/projected/ab202f23-badd-46dc-8912-de84e4218450-kube-api-access-t92sb\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:35.949777 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.949745    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:35.950956 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.950927    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:44:35.951647 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.951522    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:35.952169 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952147    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-chxht\""
Sep 16 18:44:35.952169 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952162    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:44:35.952317 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952203    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:44:35.952317 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952157    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7gjdq\""
Sep 16 18:44:35.952317 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952251    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:44:35.952317 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952293    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:44:35.952317 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952151    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g9rmr\""
Sep 16 18:44:35.952566 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952443    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:44:35.952566 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952467    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:44:35.952566 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952478    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t8xlg\""
Sep 16 18:44:35.952719 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952578    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:44:35.952719 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952676    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fgdr2\""
Sep 16 18:44:35.952848 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952799    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:44:35.952902 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952864    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:44:35.953072 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952966    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xj8nj\""
Sep 16 18:44:35.953072 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.952976    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:44:35.953072 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.953025    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:35.953072 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.953039    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:44:35.953267 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.953157    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:35.953267 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.953205    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8k2wh\""
Sep 16 18:44:35.953267 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.953249    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:44:35.954233 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.954215    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:35.954327 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:35.954281    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:44:35.954777 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.954751    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gxfwj\""
Sep 16 18:44:35.954869 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.954850    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:44:35.955817 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.955799    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:35.955898 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:35.955865    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:44:35.960855 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.960793    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cndmf\""
Sep 16 18:44:35.962743 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.962726    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:44:35.970064 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:35.970024    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:44:36.036892 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.036864    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hqjc"
Sep 16 18:44:36.037549 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.037533    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:44:36.050153 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050128    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-cni-netd\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.050232 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050168    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-ovn-node-metrics-cert\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.050232 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050202    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-run-k8s-cni-cncf-io\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.050319 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050243    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-cni-netd\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.050319 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050251    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-run-multus-certs\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.050319 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050293    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58db8d7a-7827-440d-93c3-4ef40d2cab4b-konnectivity-ca\") pod \"konnectivity-agent-mrbh7\" (UID: \"58db8d7a-7827-440d-93c3-4ef40d2cab4b\") " pod="kube-system/konnectivity-agent-mrbh7"
Sep 16 18:44:36.050319 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050314    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d07fffcd-979c-415a-83be-27c984957d2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.050485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050333    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-systemd-units\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.050485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050362    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-ovnkube-script-lib\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.050485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050391    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-registration-dir\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.050485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050398    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-systemd-units\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.050485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050416    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.050485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050441    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-var-lib-cni-multus\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.050485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050457    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-registration-dir\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.050485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050477    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-sysctl-conf\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050501    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-lib-modules\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050516    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65f8a799-79ad-437e-889d-580e125d2013-etc-tuned\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050531    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-etc-kubernetes\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050561    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050568    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-etc-selinux\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050589    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2e821ae-3c87-4fa1-8065-4004d791b3fc-kubelet-config\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050610    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65f8a799-79ad-437e-889d-580e125d2013-tmp\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050628    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm624\" (UniqueName: \"kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624\") pod \"network-check-target-47c96\" (UID: \"42e26367-9854-47ff-8046-3cd71c2b851a\") " pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050625    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050656    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-multus-daemon-config\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050695    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-run-netns\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050686    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2e821ae-3c87-4fa1-8065-4004d791b3fc-kubelet-config\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050749    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-run-netns\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050755    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-run-ovn\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050805    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-ovnkube-config\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050838    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-cnibin\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.050884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050847    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-run-ovn\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050866    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs\") pod \"network-metrics-daemon-bwr7v\" (UID: \"24c2659a-61bd-4dca-aba0-d56072325bde\") " pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050891    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-slash\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050918    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-var-lib-openvswitch\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050945    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-etc-openvswitch\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050935    2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050970    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-slash\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050975    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050994    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-var-lib-openvswitch\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.050992    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-ovnkube-script-lib\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051012    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051035    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-etc-openvswitch\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051041    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-host\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051031    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051078    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051079    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d07fffcd-979c-415a-83be-27c984957d2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051113    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-env-overrides\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.051785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051140    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92sb\" (UniqueName: \"kubernetes.io/projected/ab202f23-badd-46dc-8912-de84e4218450-kube-api-access-t92sb\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051168    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d07fffcd-979c-415a-83be-27c984957d2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051197    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-run-systemd\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051219    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-sys-fs\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051257    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-cni-binary-copy\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051274    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-run-systemd\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051318    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-hostroot\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051339    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-ovnkube-config\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051389    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-sys-fs\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051415    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58db8d7a-7827-440d-93c3-4ef40d2cab4b-agent-certs\") pod \"konnectivity-agent-mrbh7\" (UID: \"58db8d7a-7827-440d-93c3-4ef40d2cab4b\") " pod="kube-system/konnectivity-agent-mrbh7"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051452    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-log-socket\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051480    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lckmp\" (UniqueName: \"kubernetes.io/projected/41b9e54d-41ed-4362-a99a-9998cba62a60-kube-api-access-lckmp\") pod \"node-resolver-jsj4b\" (UID: \"41b9e54d-41ed-4362-a99a-9998cba62a60\") " pod="openshift-dns/node-resolver-jsj4b"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051507    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-run\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051514    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-log-socket\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051532    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d07fffcd-979c-415a-83be-27c984957d2a-cnibin\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051562    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-device-dir\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051586    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b9e54d-41ed-4362-a99a-9998cba62a60-tmp-dir\") pod \"node-resolver-jsj4b\" (UID: \"41b9e54d-41ed-4362-a99a-9998cba62a60\") " pod="openshift-dns/node-resolver-jsj4b"
Sep 16 18:44:36.052573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051612    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-var-lib-cni-bin\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051623    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-device-dir\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051638    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-multus-conf-dir\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051726    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grqpq\" (UniqueName: \"kubernetes.io/projected/de002017-1014-447e-aedd-a162b506b5b8-kube-api-access-grqpq\") pod \"iptables-alerter-n4vdz\" (UID: \"de002017-1014-447e-aedd-a162b506b5b8\") " pod="openshift-network-operator/iptables-alerter-n4vdz"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051756    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-kubernetes\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051781    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwkqm\" (UniqueName: \"kubernetes.io/projected/24c2659a-61bd-4dca-aba0-d56072325bde-kube-api-access-zwkqm\") pod \"network-metrics-daemon-bwr7v\" (UID: \"24c2659a-61bd-4dca-aba0-d56072325bde\") " pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051808    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d07fffcd-979c-415a-83be-27c984957d2a-system-cni-dir\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051831    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-var-lib-kubelet\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051859    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-node-log\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051895    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgtvx\" (UniqueName: \"kubernetes.io/projected/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-kube-api-access-kgtvx\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051921    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-node-log\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051931    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b9e54d-41ed-4362-a99a-9998cba62a60-tmp-dir\") pod \"node-resolver-jsj4b\" (UID: \"41b9e54d-41ed-4362-a99a-9998cba62a60\") " pod="openshift-dns/node-resolver-jsj4b"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051929    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-multus-cni-dir\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.051997    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-env-overrides\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052074    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-sysconfig\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052114    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-kubelet\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052152    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-kubelet\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.053271 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052153    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-run-openvswitch\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052186    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-run-openvswitch\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052199    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2e821ae-3c87-4fa1-8065-4004d791b3fc-dbus\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052227    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xn6j\" (UniqueName: \"kubernetes.io/projected/41f91dbd-52af-482f-88fe-b09f0e5c48cc-kube-api-access-6xn6j\") pod \"node-ca-mzfr8\" (UID: \"41f91dbd-52af-482f-88fe-b09f0e5c48cc\") " pod="openshift-image-registry/node-ca-mzfr8"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052264    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-system-cni-dir\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052315    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-multus-socket-dir-parent\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052340    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj8v8\" (UniqueName: \"kubernetes.io/projected/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-kube-api-access-gj8v8\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052341    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2e821ae-3c87-4fa1-8065-4004d791b3fc-dbus\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052364    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-cni-bin\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052400    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-socket-dir\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052425    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41f91dbd-52af-482f-88fe-b09f0e5c48cc-host\") pod \"node-ca-mzfr8\" (UID: \"41f91dbd-52af-482f-88fe-b09f0e5c48cc\") " pod="openshift-image-registry/node-ca-mzfr8"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052424    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-host-cni-bin\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052512    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-sysctl-d\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052519    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/41f91dbd-52af-482f-88fe-b09f0e5c48cc-host\") pod \"node-ca-mzfr8\" (UID: \"41f91dbd-52af-482f-88fe-b09f0e5c48cc\") " pod="openshift-image-registry/node-ca-mzfr8"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052517    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab202f23-badd-46dc-8912-de84e4218450-socket-dir\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052549    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-var-lib-kubelet\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052578    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvscf\" (UniqueName: \"kubernetes.io/projected/d07fffcd-979c-415a-83be-27c984957d2a-kube-api-access-zvscf\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.053972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052609    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41f91dbd-52af-482f-88fe-b09f0e5c48cc-serviceca\") pod \"node-ca-mzfr8\" (UID: \"41f91dbd-52af-482f-88fe-b09f0e5c48cc\") " pod="openshift-image-registry/node-ca-mzfr8"
Sep 16 18:44:36.054625 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052679    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d07fffcd-979c-415a-83be-27c984957d2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.054625 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052723    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-run-netns\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.054625 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052745    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de002017-1014-447e-aedd-a162b506b5b8-host-slash\") pod \"iptables-alerter-n4vdz\" (UID: \"de002017-1014-447e-aedd-a162b506b5b8\") " pod="openshift-network-operator/iptables-alerter-n4vdz"
Sep 16 18:44:36.054625 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052768    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-systemd\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.054625 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052796    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:36.054625 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052822    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41b9e54d-41ed-4362-a99a-9998cba62a60-hosts-file\") pod \"node-resolver-jsj4b\" (UID: \"41b9e54d-41ed-4362-a99a-9998cba62a60\") " pod="openshift-dns/node-resolver-jsj4b"
Sep 16 18:44:36.054625 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052860    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de002017-1014-447e-aedd-a162b506b5b8-iptables-alerter-script\") pod \"iptables-alerter-n4vdz\" (UID: \"de002017-1014-447e-aedd-a162b506b5b8\") " pod="openshift-network-operator/iptables-alerter-n4vdz"
Sep 16 18:44:36.054625 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052879    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-sys\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.054625 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052899    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptzwx\" (UniqueName: \"kubernetes.io/projected/65f8a799-79ad-437e-889d-580e125d2013-kube-api-access-ptzwx\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.054625 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052918    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-modprobe-d\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.054625 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.052940    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41b9e54d-41ed-4362-a99a-9998cba62a60-hosts-file\") pod \"node-resolver-jsj4b\" (UID: \"41b9e54d-41ed-4362-a99a-9998cba62a60\") " pod="openshift-dns/node-resolver-jsj4b"
Sep 16 18:44:36.054625 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.053092    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:36.054625 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.053211    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret podName:b2e821ae-3c87-4fa1-8065-4004d791b3fc nodeName:}" failed. No retries permitted until 2026-09-16 18:44:36.553182076 +0000 UTC m=+2.937741301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret") pod "global-pull-secret-syncer-49tcg" (UID: "b2e821ae-3c87-4fa1-8065-4004d791b3fc") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:36.054625 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.053394    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/41f91dbd-52af-482f-88fe-b09f0e5c48cc-serviceca\") pod \"node-ca-mzfr8\" (UID: \"41f91dbd-52af-482f-88fe-b09f0e5c48cc\") " pod="openshift-image-registry/node-ca-mzfr8"
Sep 16 18:44:36.055318 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.055297    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-ovn-node-metrics-cert\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.055373 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.055323    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2hqjc"
Sep 16 18:44:36.064714 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.064688    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92sb\" (UniqueName: \"kubernetes.io/projected/ab202f23-badd-46dc-8912-de84e4218450-kube-api-access-t92sb\") pod \"aws-ebs-csi-driver-node-rvv75\" (UID: \"ab202f23-badd-46dc-8912-de84e4218450\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.069253 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.069232    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lckmp\" (UniqueName: \"kubernetes.io/projected/41b9e54d-41ed-4362-a99a-9998cba62a60-kube-api-access-lckmp\") pod \"node-resolver-jsj4b\" (UID: \"41b9e54d-41ed-4362-a99a-9998cba62a60\") " pod="openshift-dns/node-resolver-jsj4b"
Sep 16 18:44:36.069705 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.069670    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xn6j\" (UniqueName: \"kubernetes.io/projected/41f91dbd-52af-482f-88fe-b09f0e5c48cc-kube-api-access-6xn6j\") pod \"node-ca-mzfr8\" (UID: \"41f91dbd-52af-482f-88fe-b09f0e5c48cc\") " pod="openshift-image-registry/node-ca-mzfr8"
Sep 16 18:44:36.075981 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.075956    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgtvx\" (UniqueName: \"kubernetes.io/projected/4e0465e6-c14b-48dc-aa48-2951cfd2e2e0-kube-api-access-kgtvx\") pod \"ovnkube-node-2rbsh\" (UID: \"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.153919 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.153856    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-cni-binary-copy\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.153919 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.153896    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-hostroot\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.154119 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.153924    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58db8d7a-7827-440d-93c3-4ef40d2cab4b-agent-certs\") pod \"konnectivity-agent-mrbh7\" (UID: \"58db8d7a-7827-440d-93c3-4ef40d2cab4b\") " pod="kube-system/konnectivity-agent-mrbh7"
Sep 16 18:44:36.154119 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.153951    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-run\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.154119 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.153975    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d07fffcd-979c-415a-83be-27c984957d2a-cnibin\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.154119 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154003    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-var-lib-cni-bin\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.154119 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154012    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-hostroot\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.154119 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154063    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-run\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.154119 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154073    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-multus-conf-dir\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.154119 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154095    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d07fffcd-979c-415a-83be-27c984957d2a-cnibin\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.154119 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154023    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-multus-conf-dir\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.154119 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154120    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grqpq\" (UniqueName: \"kubernetes.io/projected/de002017-1014-447e-aedd-a162b506b5b8-kube-api-access-grqpq\") pod \"iptables-alerter-n4vdz\" (UID: \"de002017-1014-447e-aedd-a162b506b5b8\") " pod="openshift-network-operator/iptables-alerter-n4vdz"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154141    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-kubernetes\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154168    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwkqm\" (UniqueName: \"kubernetes.io/projected/24c2659a-61bd-4dca-aba0-d56072325bde-kube-api-access-zwkqm\") pod \"network-metrics-daemon-bwr7v\" (UID: \"24c2659a-61bd-4dca-aba0-d56072325bde\") " pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154201    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d07fffcd-979c-415a-83be-27c984957d2a-system-cni-dir\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154227    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-var-lib-kubelet\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154226    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-var-lib-cni-bin\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154253    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-multus-cni-dir\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154270    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-sysconfig\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154288    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-system-cni-dir\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154303    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-multus-socket-dir-parent\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154319    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj8v8\" (UniqueName: \"kubernetes.io/projected/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-kube-api-access-gj8v8\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154336    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-sysctl-d\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154338    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-kubernetes\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154350    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-var-lib-kubelet\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154366    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvscf\" (UniqueName: \"kubernetes.io/projected/d07fffcd-979c-415a-83be-27c984957d2a-kube-api-access-zvscf\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154387    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d07fffcd-979c-415a-83be-27c984957d2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154402    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-run-netns\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.154665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154403    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-sysconfig\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154404    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d07fffcd-979c-415a-83be-27c984957d2a-system-cni-dir\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154445    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-var-lib-kubelet\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154419    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de002017-1014-447e-aedd-a162b506b5b8-host-slash\") pod \"iptables-alerter-n4vdz\" (UID: \"de002017-1014-447e-aedd-a162b506b5b8\") " pod="openshift-network-operator/iptables-alerter-n4vdz"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154474    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de002017-1014-447e-aedd-a162b506b5b8-host-slash\") pod \"iptables-alerter-n4vdz\" (UID: \"de002017-1014-447e-aedd-a162b506b5b8\") " pod="openshift-network-operator/iptables-alerter-n4vdz"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154496    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-cni-binary-copy\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154501    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-systemd\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154510    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-var-lib-kubelet\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154499    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-multus-socket-dir-parent\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154529    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-system-cni-dir\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154545    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-multus-cni-dir\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154560    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de002017-1014-447e-aedd-a162b506b5b8-iptables-alerter-script\") pod \"iptables-alerter-n4vdz\" (UID: \"de002017-1014-447e-aedd-a162b506b5b8\") " pod="openshift-network-operator/iptables-alerter-n4vdz"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154565    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-systemd\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154591    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-run-netns\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154600    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-sys\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154634    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-sysctl-d\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154676    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-sys\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154627    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptzwx\" (UniqueName: \"kubernetes.io/projected/65f8a799-79ad-437e-889d-580e125d2013-kube-api-access-ptzwx\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.155603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154718    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-modprobe-d\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154767    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-run-k8s-cni-cncf-io\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154810    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-run-k8s-cni-cncf-io\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154841    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-run-multus-certs\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154857    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-modprobe-d\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154869    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58db8d7a-7827-440d-93c3-4ef40d2cab4b-konnectivity-ca\") pod \"konnectivity-agent-mrbh7\" (UID: \"58db8d7a-7827-440d-93c3-4ef40d2cab4b\") " pod="kube-system/konnectivity-agent-mrbh7"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154896    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d07fffcd-979c-415a-83be-27c984957d2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154909    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-run-multus-certs\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154925    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-var-lib-cni-multus\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154972    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-sysctl-conf\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155005    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-lib-modules\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155032    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65f8a799-79ad-437e-889d-580e125d2013-etc-tuned\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155058    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-etc-kubernetes\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155090    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65f8a799-79ad-437e-889d-580e125d2013-tmp\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155112    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d07fffcd-979c-415a-83be-27c984957d2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155120    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d07fffcd-979c-415a-83be-27c984957d2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155126    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de002017-1014-447e-aedd-a162b506b5b8-iptables-alerter-script\") pod \"iptables-alerter-n4vdz\" (UID: \"de002017-1014-447e-aedd-a162b506b5b8\") " pod="openshift-network-operator/iptables-alerter-n4vdz"
Sep 16 18:44:36.156588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.154979    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-host-var-lib-cni-multus\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155124    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm624\" (UniqueName: \"kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624\") pod \"network-check-target-47c96\" (UID: \"42e26367-9854-47ff-8046-3cd71c2b851a\") " pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155188    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-multus-daemon-config\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155219    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-cnibin\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155242    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs\") pod \"network-metrics-daemon-bwr7v\" (UID: \"24c2659a-61bd-4dca-aba0-d56072325bde\") " pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155269    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-etc-sysctl-conf\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155271    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-host\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155313    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d07fffcd-979c-415a-83be-27c984957d2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155344    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d07fffcd-979c-415a-83be-27c984957d2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155369    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-etc-kubernetes\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155476    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-lib-modules\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155518    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58db8d7a-7827-440d-93c3-4ef40d2cab4b-konnectivity-ca\") pod \"konnectivity-agent-mrbh7\" (UID: \"58db8d7a-7827-440d-93c3-4ef40d2cab4b\") " pod="kube-system/konnectivity-agent-mrbh7"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155535    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-cnibin\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.155615    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.155668    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs podName:24c2659a-61bd-4dca-aba0-d56072325bde nodeName:}" failed. No retries permitted until 2026-09-16 18:44:36.655649863 +0000 UTC m=+3.040209093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs") pod "network-metrics-daemon-bwr7v" (UID: "24c2659a-61bd-4dca-aba0-d56072325bde") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155775    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-multus-daemon-config\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155831    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d07fffcd-979c-415a-83be-27c984957d2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.157547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.155839    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65f8a799-79ad-437e-889d-580e125d2013-host\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.158344 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.156062    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d07fffcd-979c-415a-83be-27c984957d2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.158344 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.157585    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65f8a799-79ad-437e-889d-580e125d2013-tmp\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.158344 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.157879    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58db8d7a-7827-440d-93c3-4ef40d2cab4b-agent-certs\") pod \"konnectivity-agent-mrbh7\" (UID: \"58db8d7a-7827-440d-93c3-4ef40d2cab4b\") " pod="kube-system/konnectivity-agent-mrbh7"
Sep 16 18:44:36.158344 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.157897    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65f8a799-79ad-437e-889d-580e125d2013-etc-tuned\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.168793 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.168758    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:36.168793 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.168786    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:36.168953 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.168802    2570 projected.go:196] Error preparing data for projected volume kube-api-access-bm624 for pod openshift-network-diagnostics/network-check-target-47c96: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:36.168953 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.168869    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624 podName:42e26367-9854-47ff-8046-3cd71c2b851a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:36.668851415 +0000 UTC m=+3.053410623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bm624" (UniqueName: "kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624") pod "network-check-target-47c96" (UID: "42e26367-9854-47ff-8046-3cd71c2b851a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:36.168953 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.168874    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvscf\" (UniqueName: \"kubernetes.io/projected/d07fffcd-979c-415a-83be-27c984957d2a-kube-api-access-zvscf\") pod \"multus-additional-cni-plugins-pbd6r\" (UID: \"d07fffcd-979c-415a-83be-27c984957d2a\") " pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.177746 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.177721    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj8v8\" (UniqueName: \"kubernetes.io/projected/fe91104f-cafe-4144-9f6f-c8fd9ec3e45a-kube-api-access-gj8v8\") pod \"multus-56tqs\" (UID: \"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a\") " pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.178039 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.178019    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwkqm\" (UniqueName: \"kubernetes.io/projected/24c2659a-61bd-4dca-aba0-d56072325bde-kube-api-access-zwkqm\") pod \"network-metrics-daemon-bwr7v\" (UID: \"24c2659a-61bd-4dca-aba0-d56072325bde\") " pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:36.178134 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.178114    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grqpq\" (UniqueName: \"kubernetes.io/projected/de002017-1014-447e-aedd-a162b506b5b8-kube-api-access-grqpq\") pod \"iptables-alerter-n4vdz\" (UID: \"de002017-1014-447e-aedd-a162b506b5b8\") " pod="openshift-network-operator/iptables-alerter-n4vdz"
Sep 16 18:44:36.184184 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.184160    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptzwx\" (UniqueName: \"kubernetes.io/projected/65f8a799-79ad-437e-889d-580e125d2013-kube-api-access-ptzwx\") pod \"tuned-hnwrq\" (UID: \"65f8a799-79ad-437e-889d-580e125d2013\") " pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.247322 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.247298    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:44:36.257682 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.257658    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75"
Sep 16 18:44:36.267036 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.267009    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jsj4b"
Sep 16 18:44:36.274869 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.274835    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mzfr8"
Sep 16 18:44:36.282677 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.282658    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-56tqs"
Sep 16 18:44:36.291510 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.291491    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n4vdz"
Sep 16 18:44:36.299153 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.299130    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mrbh7"
Sep 16 18:44:36.308453 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.308433    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hnwrq"
Sep 16 18:44:36.313817 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.313798    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pbd6r"
Sep 16 18:44:36.557931 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.557679    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:36.558103 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.557813    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:36.558103 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.558043    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret podName:b2e821ae-3c87-4fa1-8065-4004d791b3fc nodeName:}" failed. No retries permitted until 2026-09-16 18:44:37.558018523 +0000 UTC m=+3.942577732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret") pod "global-pull-secret-syncer-49tcg" (UID: "b2e821ae-3c87-4fa1-8065-4004d791b3fc") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:36.592028 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.592000    2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:36.658866 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.658839    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs\") pod \"network-metrics-daemon-bwr7v\" (UID: \"24c2659a-61bd-4dca-aba0-d56072325bde\") " pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:36.659044 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.658964    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:36.659044 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.659032    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs podName:24c2659a-61bd-4dca-aba0-d56072325bde nodeName:}" failed. No retries permitted until 2026-09-16 18:44:37.659012544 +0000 UTC m=+4.043571750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs") pod "network-metrics-daemon-bwr7v" (UID: "24c2659a-61bd-4dca-aba0-d56072325bde") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:36.759438 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:36.759393    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm624\" (UniqueName: \"kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624\") pod \"network-check-target-47c96\" (UID: \"42e26367-9854-47ff-8046-3cd71c2b851a\") " pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:36.759643 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.759576    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:36.759643 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.759613    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:36.759643 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.759632    2570 projected.go:196] Error preparing data for projected volume kube-api-access-bm624 for pod openshift-network-diagnostics/network-check-target-47c96: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:36.759865 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:36.759739    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624 podName:42e26367-9854-47ff-8046-3cd71c2b851a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:37.759717531 +0000 UTC m=+4.144276772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm624" (UniqueName: "kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624") pod "network-check-target-47c96" (UID: "42e26367-9854-47ff-8046-3cd71c2b851a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:37.058078 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.058013    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:36 +0000 UTC" deadline="2028-02-27 23:17:37.534206697 +0000 UTC"
Sep 16 18:44:37.058078 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.058064    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12700h33m0.476145955s"
Sep 16 18:44:37.079269 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:37.079242    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f8a799_79ad_437e_889d_580e125d2013.slice/crio-31eb93d7710b60af90b442bb15769afbb4cbb0136259312a53289c6953bdbbc9 WatchSource:0}: Error finding container 31eb93d7710b60af90b442bb15769afbb4cbb0136259312a53289c6953bdbbc9: Status 404 returned error can't find the container with id 31eb93d7710b60af90b442bb15769afbb4cbb0136259312a53289c6953bdbbc9
Sep 16 18:44:37.081132 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:37.081103    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07fffcd_979c_415a_83be_27c984957d2a.slice/crio-3dcf4f7ba0134dd178cc2c43c972a62e23b1f1c5346b0c480ae08108cfa5551c WatchSource:0}: Error finding container 3dcf4f7ba0134dd178cc2c43c972a62e23b1f1c5346b0c480ae08108cfa5551c: Status 404 returned error can't find the container with id 3dcf4f7ba0134dd178cc2c43c972a62e23b1f1c5346b0c480ae08108cfa5551c
Sep 16 18:44:37.082080 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:37.081821    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe91104f_cafe_4144_9f6f_c8fd9ec3e45a.slice/crio-bdc1b38824e2c5278b8b08796e3efba7826f188825cb99745ff106d20f471175 WatchSource:0}: Error finding container bdc1b38824e2c5278b8b08796e3efba7826f188825cb99745ff106d20f471175: Status 404 returned error can't find the container with id bdc1b38824e2c5278b8b08796e3efba7826f188825cb99745ff106d20f471175
Sep 16 18:44:37.085087 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:37.085068    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58db8d7a_7827_440d_93c3_4ef40d2cab4b.slice/crio-dbf02eca1d39eca99af6145e2ad23b2ebd21168854e4e700920896001f8a0cac WatchSource:0}: Error finding container dbf02eca1d39eca99af6145e2ad23b2ebd21168854e4e700920896001f8a0cac: Status 404 returned error can't find the container with id dbf02eca1d39eca99af6145e2ad23b2ebd21168854e4e700920896001f8a0cac
Sep 16 18:44:37.086211 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:37.086187    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab202f23_badd_46dc_8912_de84e4218450.slice/crio-dffb09b3927b2ae02540e7fd8226dec1dd35fceec2501b8ba8a268f6947412a4 WatchSource:0}: Error finding container dffb09b3927b2ae02540e7fd8226dec1dd35fceec2501b8ba8a268f6947412a4: Status 404 returned error can't find the container with id dffb09b3927b2ae02540e7fd8226dec1dd35fceec2501b8ba8a268f6947412a4
Sep 16 18:44:37.088803 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:37.088030    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f91dbd_52af_482f_88fe_b09f0e5c48cc.slice/crio-0ddaa39a4e14737f7a1d45715a681a36dd3d06f95f1483f3fa8c80358b83187c WatchSource:0}: Error finding container 0ddaa39a4e14737f7a1d45715a681a36dd3d06f95f1483f3fa8c80358b83187c: Status 404 returned error can't find the container with id 0ddaa39a4e14737f7a1d45715a681a36dd3d06f95f1483f3fa8c80358b83187c
Sep 16 18:44:37.088803 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:37.088380    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b9e54d_41ed_4362_a99a_9998cba62a60.slice/crio-0c0708f66712f501553e0cf3e64925bce24e7f111b79628e83fe7ca385595f54 WatchSource:0}: Error finding container 0c0708f66712f501553e0cf3e64925bce24e7f111b79628e83fe7ca385595f54: Status 404 returned error can't find the container with id 0c0708f66712f501553e0cf3e64925bce24e7f111b79628e83fe7ca385595f54
Sep 16 18:44:37.090000 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:37.089702    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0465e6_c14b_48dc_aa48_2951cfd2e2e0.slice/crio-fa33e0326fb61e77a0f5c999c242235e2119507be97a2bdb53fa964c4c2c4fe0 WatchSource:0}: Error finding container fa33e0326fb61e77a0f5c999c242235e2119507be97a2bdb53fa964c4c2c4fe0: Status 404 returned error can't find the container with id fa33e0326fb61e77a0f5c999c242235e2119507be97a2bdb53fa964c4c2c4fe0
Sep 16 18:44:37.090984 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:44:37.090900    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde002017_1014_447e_aedd_a162b506b5b8.slice/crio-e8a26bff9037b61fa78794e402b82996b70644d006e31311f711963365a7a990 WatchSource:0}: Error finding container e8a26bff9037b61fa78794e402b82996b70644d006e31311f711963365a7a990: Status 404 returned error can't find the container with id e8a26bff9037b61fa78794e402b82996b70644d006e31311f711963365a7a990
Sep 16 18:44:37.156443 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.156419    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:37.156556 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:37.156534    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:44:37.163950 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.163921    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh" event={"ID":"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0","Type":"ContainerStarted","Data":"fa33e0326fb61e77a0f5c999c242235e2119507be97a2bdb53fa964c4c2c4fe0"}
Sep 16 18:44:37.165519 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.165493    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mzfr8" event={"ID":"41f91dbd-52af-482f-88fe-b09f0e5c48cc","Type":"ContainerStarted","Data":"0ddaa39a4e14737f7a1d45715a681a36dd3d06f95f1483f3fa8c80358b83187c"}
Sep 16 18:44:37.169625 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.169602    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jsj4b" event={"ID":"41b9e54d-41ed-4362-a99a-9998cba62a60","Type":"ContainerStarted","Data":"0c0708f66712f501553e0cf3e64925bce24e7f111b79628e83fe7ca385595f54"}
Sep 16 18:44:37.171178 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.171153    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mrbh7" event={"ID":"58db8d7a-7827-440d-93c3-4ef40d2cab4b","Type":"ContainerStarted","Data":"dbf02eca1d39eca99af6145e2ad23b2ebd21168854e4e700920896001f8a0cac"}
Sep 16 18:44:37.172328 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.172307    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-56tqs" event={"ID":"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a","Type":"ContainerStarted","Data":"bdc1b38824e2c5278b8b08796e3efba7826f188825cb99745ff106d20f471175"}
Sep 16 18:44:37.175137 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.175113    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbd6r" event={"ID":"d07fffcd-979c-415a-83be-27c984957d2a","Type":"ContainerStarted","Data":"3dcf4f7ba0134dd178cc2c43c972a62e23b1f1c5346b0c480ae08108cfa5551c"}
Sep 16 18:44:37.176606 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.176578    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75" event={"ID":"ab202f23-badd-46dc-8912-de84e4218450","Type":"ContainerStarted","Data":"dffb09b3927b2ae02540e7fd8226dec1dd35fceec2501b8ba8a268f6947412a4"}
Sep 16 18:44:37.178075 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.178026    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n4vdz" event={"ID":"de002017-1014-447e-aedd-a162b506b5b8","Type":"ContainerStarted","Data":"e8a26bff9037b61fa78794e402b82996b70644d006e31311f711963365a7a990"}
Sep 16 18:44:37.179363 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.179344    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hnwrq" event={"ID":"65f8a799-79ad-437e-889d-580e125d2013","Type":"ContainerStarted","Data":"31eb93d7710b60af90b442bb15769afbb4cbb0136259312a53289c6953bdbbc9"}
Sep 16 18:44:37.564784 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.564671    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:37.565056 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:37.564846    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:37.565056 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:37.564906    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret podName:b2e821ae-3c87-4fa1-8065-4004d791b3fc nodeName:}" failed. No retries permitted until 2026-09-16 18:44:39.564888334 +0000 UTC m=+5.949447556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret") pod "global-pull-secret-syncer-49tcg" (UID: "b2e821ae-3c87-4fa1-8065-4004d791b3fc") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:37.666075 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.666039    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs\") pod \"network-metrics-daemon-bwr7v\" (UID: \"24c2659a-61bd-4dca-aba0-d56072325bde\") " pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:37.666255 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:37.666228    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:37.666317 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:37.666292    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs podName:24c2659a-61bd-4dca-aba0-d56072325bde nodeName:}" failed. No retries permitted until 2026-09-16 18:44:39.666269693 +0000 UTC m=+6.050828903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs") pod "network-metrics-daemon-bwr7v" (UID: "24c2659a-61bd-4dca-aba0-d56072325bde") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:37.766843 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:37.766805    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm624\" (UniqueName: \"kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624\") pod \"network-check-target-47c96\" (UID: \"42e26367-9854-47ff-8046-3cd71c2b851a\") " pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:37.767037 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:37.767018    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:37.767107 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:37.767045    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:37.767107 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:37.767060    2570 projected.go:196] Error preparing data for projected volume kube-api-access-bm624 for pod openshift-network-diagnostics/network-check-target-47c96: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:37.767209 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:37.767118    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624 podName:42e26367-9854-47ff-8046-3cd71c2b851a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:39.767099801 +0000 UTC m=+6.151659009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm624" (UniqueName: "kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624") pod "network-check-target-47c96" (UID: "42e26367-9854-47ff-8046-3cd71c2b851a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:38.058293 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:38.058250    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:36 +0000 UTC" deadline="2028-04-21 17:06:29.872717536 +0000 UTC"
Sep 16 18:44:38.058293 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:38.058289    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13990h21m51.814433692s"
Sep 16 18:44:38.156909 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:38.156881    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:38.157083 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:38.157027    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:44:38.157509 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:38.157483    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:38.157649 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:38.157625    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:44:38.195956 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:38.195323    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal" event={"ID":"f223e05d53cb9f1d1ecb96b3f640ab21","Type":"ContainerStarted","Data":"c3881cea427c6619a5b459d362b439259790381fffa6fe9f15d26dabf6cff65d"}
Sep 16 18:44:38.198859 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:38.198828    2570 generic.go:356] "Generic (PLEG): container finished" podID="8fd94c484fe2786e35424aca732ed6cf" containerID="57ca1178388143035684562b6512cac7f8522ad399adca9d52e0ac527a820e94" exitCode=0
Sep 16 18:44:38.198966 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:38.198871    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal" event={"ID":"8fd94c484fe2786e35424aca732ed6cf","Type":"ContainerDied","Data":"57ca1178388143035684562b6512cac7f8522ad399adca9d52e0ac527a820e94"}
Sep 16 18:44:38.241732 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:38.241650    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-244.ec2.internal" podStartSLOduration=3.24162964 podStartE2EDuration="3.24162964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:38.222471291 +0000 UTC m=+4.607030519" watchObservedRunningTime="2026-09-16 18:44:38.24162964 +0000 UTC m=+4.626188867"
Sep 16 18:44:39.155924 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:39.155392    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:39.155924 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:39.155522    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:44:39.221655 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:39.221585    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal" event={"ID":"8fd94c484fe2786e35424aca732ed6cf","Type":"ContainerStarted","Data":"252ddaa46857ee744305e3307065290fada061eae1a9d8c730eea7266f3edb3d"}
Sep 16 18:44:39.583340 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:39.583298    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:39.583541 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:39.583439    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:39.583541 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:39.583501    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret podName:b2e821ae-3c87-4fa1-8065-4004d791b3fc nodeName:}" failed. No retries permitted until 2026-09-16 18:44:43.583481974 +0000 UTC m=+9.968041180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret") pod "global-pull-secret-syncer-49tcg" (UID: "b2e821ae-3c87-4fa1-8065-4004d791b3fc") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:39.684646 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:39.684603    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs\") pod \"network-metrics-daemon-bwr7v\" (UID: \"24c2659a-61bd-4dca-aba0-d56072325bde\") " pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:39.684834 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:39.684761    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:39.684922 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:39.684848    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs podName:24c2659a-61bd-4dca-aba0-d56072325bde nodeName:}" failed. No retries permitted until 2026-09-16 18:44:43.684827716 +0000 UTC m=+10.069386922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs") pod "network-metrics-daemon-bwr7v" (UID: "24c2659a-61bd-4dca-aba0-d56072325bde") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:39.786011 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:39.785974    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm624\" (UniqueName: \"kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624\") pod \"network-check-target-47c96\" (UID: \"42e26367-9854-47ff-8046-3cd71c2b851a\") " pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:39.786221 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:39.786171    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:39.786221 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:39.786198    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:39.786221 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:39.786211    2570 projected.go:196] Error preparing data for projected volume kube-api-access-bm624 for pod openshift-network-diagnostics/network-check-target-47c96: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:39.786402 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:39.786268    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624 podName:42e26367-9854-47ff-8046-3cd71c2b851a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:43.786247969 +0000 UTC m=+10.170807186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm624" (UniqueName: "kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624") pod "network-check-target-47c96" (UID: "42e26367-9854-47ff-8046-3cd71c2b851a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:40.158526 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:40.158492    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:40.159057 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:40.158645    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:44:40.159132 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:40.159102    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:40.159243 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:40.159201    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:44:41.156172 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:41.156138    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:41.156377 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:41.156289    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:44:42.156877 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:42.156835    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:42.157351 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:42.156980    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:44:42.157522 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:42.157491    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:42.157620 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:42.157597    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:44:43.156332 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:43.155826    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:43.156332 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:43.155969    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:44:43.619675 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:43.619634    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:43.620166 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:43.619863    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:43.620166 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:43.619940    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret podName:b2e821ae-3c87-4fa1-8065-4004d791b3fc nodeName:}" failed. No retries permitted until 2026-09-16 18:44:51.619919444 +0000 UTC m=+18.004478666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret") pod "global-pull-secret-syncer-49tcg" (UID: "b2e821ae-3c87-4fa1-8065-4004d791b3fc") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:43.720778 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:43.720733    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs\") pod \"network-metrics-daemon-bwr7v\" (UID: \"24c2659a-61bd-4dca-aba0-d56072325bde\") " pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:43.720947 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:43.720903    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:43.721023 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:43.720987    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs podName:24c2659a-61bd-4dca-aba0-d56072325bde nodeName:}" failed. No retries permitted until 2026-09-16 18:44:51.720964457 +0000 UTC m=+18.105523677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs") pod "network-metrics-daemon-bwr7v" (UID: "24c2659a-61bd-4dca-aba0-d56072325bde") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:43.822105 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:43.822073    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm624\" (UniqueName: \"kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624\") pod \"network-check-target-47c96\" (UID: \"42e26367-9854-47ff-8046-3cd71c2b851a\") " pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:43.822304 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:43.822249    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:43.822304 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:43.822270    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:43.822304 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:43.822283    2570 projected.go:196] Error preparing data for projected volume kube-api-access-bm624 for pod openshift-network-diagnostics/network-check-target-47c96: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:43.822466 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:43.822343    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624 podName:42e26367-9854-47ff-8046-3cd71c2b851a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:51.822322492 +0000 UTC m=+18.206881715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm624" (UniqueName: "kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624") pod "network-check-target-47c96" (UID: "42e26367-9854-47ff-8046-3cd71c2b851a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:44.157581 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:44.157013    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:44.157581 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:44.157135    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:44:44.157581 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:44.157188    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:44.157581 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:44.157257    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:44:45.155898 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:45.155866    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:45.156339 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:45.155987    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:44:46.155697 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:46.155652    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:46.155878 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:46.155798    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:44:46.155878 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:46.155860    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:46.156330 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:46.155981    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:44:47.156059 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:47.156032    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:47.156456 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:47.156132    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:44:48.155383 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:48.155303    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:48.155526 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:48.155448    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:44:48.155597 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:48.155532    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:48.155662 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:48.155641    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:44:49.155850 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:49.155817    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:49.156275 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:49.155936    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:44:50.155475 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:50.155404    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:50.155475 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:50.155425    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:50.155781 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:50.155541    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:44:50.155781 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:50.155715    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:44:51.155939 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:51.155908    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:51.156370 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:51.156007    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:44:51.680879 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:51.680851    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:51.681038 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:51.680973    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:51.681038 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:51.681019    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret podName:b2e821ae-3c87-4fa1-8065-4004d791b3fc nodeName:}" failed. No retries permitted until 2026-09-16 18:45:07.681006847 +0000 UTC m=+34.065566060 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret") pod "global-pull-secret-syncer-49tcg" (UID: "b2e821ae-3c87-4fa1-8065-4004d791b3fc") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:51.782081 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:51.782055    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs\") pod \"network-metrics-daemon-bwr7v\" (UID: \"24c2659a-61bd-4dca-aba0-d56072325bde\") " pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:51.782201 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:51.782144    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:51.782201 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:51.782188    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs podName:24c2659a-61bd-4dca-aba0-d56072325bde nodeName:}" failed. No retries permitted until 2026-09-16 18:45:07.782174475 +0000 UTC m=+34.166733675 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs") pod "network-metrics-daemon-bwr7v" (UID: "24c2659a-61bd-4dca-aba0-d56072325bde") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:51.882820 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:51.882787    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm624\" (UniqueName: \"kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624\") pod \"network-check-target-47c96\" (UID: \"42e26367-9854-47ff-8046-3cd71c2b851a\") " pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:51.882987 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:51.882968    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:51.883031 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:51.882993    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:51.883031 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:51.883004    2570 projected.go:196] Error preparing data for projected volume kube-api-access-bm624 for pod openshift-network-diagnostics/network-check-target-47c96: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:51.883125 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:51.883052    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624 podName:42e26367-9854-47ff-8046-3cd71c2b851a nodeName:}" failed. No retries permitted until 2026-09-16 18:45:07.883039287 +0000 UTC m=+34.267598501 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm624" (UniqueName: "kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624") pod "network-check-target-47c96" (UID: "42e26367-9854-47ff-8046-3cd71c2b851a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:52.155851 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:52.155814    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:52.156012 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:52.155824    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:52.156012 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:52.155952    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:44:52.156381 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:52.156058    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:44:53.155578 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:53.155544    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:53.155794 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:53.155659    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:44:54.156331 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:54.156302    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:54.156784 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:54.156403    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:44:54.156784 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:54.156524    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:54.156784 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:54.156669    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:44:55.155984 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:55.155964    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:55.156108 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:55.156083    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:44:55.257771 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:55.257343    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hnwrq" event={"ID":"65f8a799-79ad-437e-889d-580e125d2013","Type":"ContainerStarted","Data":"ede5ffbac3679d88e444b5cce931c671b39e313f0df6d4df34316e7672d63534"}
Sep 16 18:44:55.278289 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:55.278233    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-244.ec2.internal" podStartSLOduration=20.27821309 podStartE2EDuration="20.27821309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:39.239972242 +0000 UTC m=+5.624531472" watchObservedRunningTime="2026-09-16 18:44:55.27821309 +0000 UTC m=+21.662772314"
Sep 16 18:44:56.156243 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.156086    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:56.156363 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.156154    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:56.156363 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:56.156313    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:44:56.156429 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:56.156382    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:44:56.261451 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.261429    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh" event={"ID":"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0","Type":"ContainerStarted","Data":"33f46b4c0f76fd52250f9d750134cee66d0d0a0f7b60904bca6daebcf0d0acaa"}
Sep 16 18:44:56.262154 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.261499    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh" event={"ID":"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0","Type":"ContainerStarted","Data":"3c77fe1637267f45e96352f851e5a89e703d7119ae986eff3700e4566598acf3"}
Sep 16 18:44:56.262154 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.261547    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh" event={"ID":"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0","Type":"ContainerStarted","Data":"e5ca5a48bd18ed46bbb1db38a56cbbea7ee23f5deb06c59768a9f29bcf7271da"}
Sep 16 18:44:56.262154 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.261576    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh" event={"ID":"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0","Type":"ContainerStarted","Data":"260f142bcb4d311eee8c5cbe4ab97f77241c938d6a1228e7f5f9dd8524407a29"}
Sep 16 18:44:56.262154 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.261603    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh" event={"ID":"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0","Type":"ContainerStarted","Data":"8ad82e7a719e9cab45b6b4df39921c0c981e2c9ed0da39b8e2d354afc27411ca"}
Sep 16 18:44:56.262154 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.261636    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh" event={"ID":"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0","Type":"ContainerStarted","Data":"0934591e02c5a37baee78c5c240b61e993bc186db07d76dea6de062a852cb20a"}
Sep 16 18:44:56.262769 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.262744    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mzfr8" event={"ID":"41f91dbd-52af-482f-88fe-b09f0e5c48cc","Type":"ContainerStarted","Data":"b63eca9f288f1d2809eaf667b8769c1b52a8309b49a56289c679afeb739a6326"}
Sep 16 18:44:56.264101 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.264082    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jsj4b" event={"ID":"41b9e54d-41ed-4362-a99a-9998cba62a60","Type":"ContainerStarted","Data":"af06fa785f15cb147b2f74a9e982d6065b674d58eddb985e6d2a3b29fd0d3b95"}
Sep 16 18:44:56.265453 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.265430    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mrbh7" event={"ID":"58db8d7a-7827-440d-93c3-4ef40d2cab4b","Type":"ContainerStarted","Data":"830abea1a08a8e880a26975c6b1ef2169523aa990917bd9c57b77f86b68b33c9"}
Sep 16 18:44:56.266824 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.266803    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-56tqs" event={"ID":"fe91104f-cafe-4144-9f6f-c8fd9ec3e45a","Type":"ContainerStarted","Data":"c3325205d8ddd3b2852b386e4eb22b288ca19e8d685b3cdd462947099860f0cf"}
Sep 16 18:44:56.268136 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.268118    2570 generic.go:356] "Generic (PLEG): container finished" podID="d07fffcd-979c-415a-83be-27c984957d2a" containerID="475d93b87c3cd13fa8b3b6a8b2d28a93741069c182a4d13adae9c541c8bd5760" exitCode=0
Sep 16 18:44:56.268229 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.268161    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbd6r" event={"ID":"d07fffcd-979c-415a-83be-27c984957d2a","Type":"ContainerDied","Data":"475d93b87c3cd13fa8b3b6a8b2d28a93741069c182a4d13adae9c541c8bd5760"}
Sep 16 18:44:56.269594 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.269576    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75" event={"ID":"ab202f23-badd-46dc-8912-de84e4218450","Type":"ContainerStarted","Data":"93d613ae831293e1d9a4f6db2d6aa96b9b7a7ef8931dfba1168195d1149a1b8c"}
Sep 16 18:44:56.279424 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.279390    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hnwrq" podStartSLOduration=3.266331381 podStartE2EDuration="21.279378496s" totalImagesPullingTime="18.013047115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:55.278646448 +0000 UTC m=+21.663205674" watchObservedRunningTime="2026-09-16 18:44:56.279378496 +0000 UTC m=+22.663937718"
Sep 16 18:44:56.279860 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.279836    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mzfr8" podStartSLOduration=3.277685965 podStartE2EDuration="21.279830099s" totalImagesPullingTime="18.002144134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:56.279244057 +0000 UTC m=+22.663803281" watchObservedRunningTime="2026-09-16 18:44:56.279830099 +0000 UTC m=+22.664389322"
Sep 16 18:44:56.299105 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.299068    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-56tqs" podStartSLOduration=3.223827175 podStartE2EDuration="21.299054598s" totalImagesPullingTime="18.075227423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:56.29875693 +0000 UTC m=+22.683316153" watchObservedRunningTime="2026-09-16 18:44:56.299054598 +0000 UTC m=+22.683613821"
Sep 16 18:44:56.341758 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.341715    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jsj4b" podStartSLOduration=3.342701297 podStartE2EDuration="21.341701846s" totalImagesPullingTime="17.999000549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:56.34164714 +0000 UTC m=+22.726206364" watchObservedRunningTime="2026-09-16 18:44:56.341701846 +0000 UTC m=+22.726261070"
Sep 16 18:44:56.341974 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.341952    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mrbh7" podStartSLOduration=10.421334953 podStartE2EDuration="21.341940818s" totalImagesPullingTime="10.920605865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:56.324375115 +0000 UTC m=+22.708934331" watchObservedRunningTime="2026-09-16 18:44:56.341940818 +0000 UTC m=+22.726500042"
Sep 16 18:44:56.518934 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:56.518913    2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:44:57.153778 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:57.153655    2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:44:56.518930557Z","UUID":"a98a8874-75d9-4189-a8fe-4940df2508a3","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:44:57.155390 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:57.155359    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:44:57.155390 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:57.155392    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:44:57.155552 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:57.155424    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:57.155552 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:57.155518    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:44:57.274750 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:57.274707    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75" event={"ID":"ab202f23-badd-46dc-8912-de84e4218450","Type":"ContainerStarted","Data":"e13e4dc24fc4a4f80a90a5d6aa0cda760f57c379b909a75abe023e25d517369a"}
Sep 16 18:44:57.276511 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:57.276484    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n4vdz" event={"ID":"de002017-1014-447e-aedd-a162b506b5b8","Type":"ContainerStarted","Data":"67e8b5592a5b8b3f2b95c228ed75321b7f1dfe9a954a49900b341a5b166fb729"}
Sep 16 18:44:58.156398 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:58.156317    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:44:58.156644 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:58.156452    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:44:58.156644 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:58.156502    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:44:58.156644 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:58.156630    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:44:58.281612 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:58.281571    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75" event={"ID":"ab202f23-badd-46dc-8912-de84e4218450","Type":"ContainerStarted","Data":"0c8840a8a66faf2a54742cb01cd43698ebd01009bf7ae523c4b9211bcb35dcfc"}
Sep 16 18:44:58.285573 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:58.285544    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh" event={"ID":"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0","Type":"ContainerStarted","Data":"90c9c0feb4fd4c0418704329a6d2830719cdccf91cf38c4863f469466633ddb2"}
Sep 16 18:44:58.302015 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:58.301966    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rvv75" podStartSLOduration=3.18437692 podStartE2EDuration="23.301941965s" totalImagesPullingTime="20.117565045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:58.301399693 +0000 UTC m=+24.685958916" watchObservedRunningTime="2026-09-16 18:44:58.301941965 +0000 UTC m=+24.686501185"
Sep 16 18:44:58.302278 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:58.302251    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n4vdz" podStartSLOduration=5.302913554 podStartE2EDuration="23.302244011s" totalImagesPullingTime="17.999330457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:57.297703416 +0000 UTC m=+23.682262639" watchObservedRunningTime="2026-09-16 18:44:58.302244011 +0000 UTC m=+24.686803234"
Sep 16 18:44:58.436243 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:58.436143    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mrbh7"
Sep 16 18:44:59.155636 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:44:59.155600    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:44:59.155827 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:44:59.155738    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:45:00.156258 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:00.156101    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:45:00.156566 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:00.156101    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:45:00.156566 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:00.156302    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:45:00.156566 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:00.156368    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:45:00.234974 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:00.234947    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mrbh7"
Sep 16 18:45:00.235443 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:00.235423    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mrbh7"
Sep 16 18:45:00.294206 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:00.294174    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh" event={"ID":"4e0465e6-c14b-48dc-aa48-2951cfd2e2e0","Type":"ContainerStarted","Data":"211d7011f9b0a66a1f1fbf7d21d61e8ff79832fe382752b3e36075c07f441bda"}
Sep 16 18:45:00.294514 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:00.294490    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:45:00.295037 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:00.294983    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mrbh7"
Sep 16 18:45:00.309001 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:00.308982    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:45:00.328075 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:00.328027    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh" podStartSLOduration=7.262774464 podStartE2EDuration="25.327974877s" totalImagesPullingTime="18.065200413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:00.327175042 +0000 UTC m=+26.711734264" watchObservedRunningTime="2026-09-16 18:45:00.327974877 +0000 UTC m=+26.712534103"
Sep 16 18:45:01.155812 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:01.155786    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:45:01.155987 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:01.155869    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:45:01.298406 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:01.298370    2570 generic.go:356] "Generic (PLEG): container finished" podID="d07fffcd-979c-415a-83be-27c984957d2a" containerID="991dfc4e81cb487965a1a838478aa3f102bad32f6777df6e20a89c15e911b8ae" exitCode=0
Sep 16 18:45:01.299371 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:01.298455    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbd6r" event={"ID":"d07fffcd-979c-415a-83be-27c984957d2a","Type":"ContainerDied","Data":"991dfc4e81cb487965a1a838478aa3f102bad32f6777df6e20a89c15e911b8ae"}
Sep 16 18:45:01.299371 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:01.298931    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:45:01.299371 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:01.299003    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:45:01.313867 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:01.313849    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:45:02.156322 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:02.156295    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:45:02.156463 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:02.156425    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:45:02.156506 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:02.156472    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:45:02.156581 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:02.156564    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:45:02.196163 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:02.196136    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-49tcg"]
Sep 16 18:45:02.199902 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:02.199879    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-47c96"]
Sep 16 18:45:02.200034 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:02.199946    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:45:02.200034 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:02.200012    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:45:02.200486 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:02.200466    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bwr7v"]
Sep 16 18:45:02.299958 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:02.299934    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:45:02.300370 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:02.299943    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:45:02.300370 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:02.300054    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:45:02.300370 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:02.300096    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:45:03.303998 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:03.303960    2570 generic.go:356] "Generic (PLEG): container finished" podID="d07fffcd-979c-415a-83be-27c984957d2a" containerID="9167df1ad8031fff82903af4b7fc422ba0b1c407da04ad128483fa1be969f28d" exitCode=0
Sep 16 18:45:03.304435 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:03.304050    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbd6r" event={"ID":"d07fffcd-979c-415a-83be-27c984957d2a","Type":"ContainerDied","Data":"9167df1ad8031fff82903af4b7fc422ba0b1c407da04ad128483fa1be969f28d"}
Sep 16 18:45:04.158953 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:04.158924    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:45:04.159085 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:04.159021    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:45:04.159085 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:04.159045    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:45:04.159085 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:04.159037    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:45:04.159241 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:04.159114    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:45:04.159241 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:04.159195    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:45:04.308358 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:04.308330    2570 generic.go:356] "Generic (PLEG): container finished" podID="d07fffcd-979c-415a-83be-27c984957d2a" containerID="49573abb050eb7da20769dac4c002d6ba5b8a5c88baf2fc4898699a0d791a7e9" exitCode=0
Sep 16 18:45:04.308713 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:04.308383    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbd6r" event={"ID":"d07fffcd-979c-415a-83be-27c984957d2a","Type":"ContainerDied","Data":"49573abb050eb7da20769dac4c002d6ba5b8a5c88baf2fc4898699a0d791a7e9"}
Sep 16 18:45:06.160407 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:06.160307    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:45:06.160407 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:06.160317    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:45:06.160407 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:06.160429    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-49tcg" podUID="b2e821ae-3c87-4fa1-8065-4004d791b3fc"
Sep 16 18:45:06.161137 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:06.160596    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:45:06.161137 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:06.160579    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bwr7v" podUID="24c2659a-61bd-4dca-aba0-d56072325bde"
Sep 16 18:45:06.161137 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:06.160699    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-47c96" podUID="42e26367-9854-47ff-8046-3cd71c2b851a"
Sep 16 18:45:07.703495 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:07.703447    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:45:07.703898 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:07.703602    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:07.703898 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:07.703678    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret podName:b2e821ae-3c87-4fa1-8065-4004d791b3fc nodeName:}" failed. No retries permitted until 2026-09-16 18:45:39.703660171 +0000 UTC m=+66.088219381 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret") pod "global-pull-secret-syncer-49tcg" (UID: "b2e821ae-3c87-4fa1-8065-4004d791b3fc") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:07.803805 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:07.803773    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs\") pod \"network-metrics-daemon-bwr7v\" (UID: \"24c2659a-61bd-4dca-aba0-d56072325bde\") " pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:45:07.803961 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:07.803889    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:07.803961 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:07.803941    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs podName:24c2659a-61bd-4dca-aba0-d56072325bde nodeName:}" failed. No retries permitted until 2026-09-16 18:45:39.803926691 +0000 UTC m=+66.188485905 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs") pod "network-metrics-daemon-bwr7v" (UID: "24c2659a-61bd-4dca-aba0-d56072325bde") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:07.905042 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:07.904973    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm624\" (UniqueName: \"kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624\") pod \"network-check-target-47c96\" (UID: \"42e26367-9854-47ff-8046-3cd71c2b851a\") " pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:45:07.905165 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:07.905114    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:07.905165 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:07.905129    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:07.905165 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:07.905137    2570 projected.go:196] Error preparing data for projected volume kube-api-access-bm624 for pod openshift-network-diagnostics/network-check-target-47c96: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:07.905279 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:07.905181    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624 podName:42e26367-9854-47ff-8046-3cd71c2b851a nodeName:}" failed. No retries permitted until 2026-09-16 18:45:39.905168937 +0000 UTC m=+66.289728152 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm624" (UniqueName: "kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624") pod "network-check-target-47c96" (UID: "42e26367-9854-47ff-8046-3cd71c2b851a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:07.920254 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:07.920231    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-244.ec2.internal" event="NodeReady"
Sep 16 18:45:07.920368 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:07.920353    2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:45:07.971798 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:07.971768    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"]
Sep 16 18:45:08.002931 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.002909    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"]
Sep 16 18:45:08.003063 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.003046    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.006466 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.006367    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:45:08.006568 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.006480    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:45:08.006883 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.006865    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:45:08.008571 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.008415    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cnz8v\""
Sep 16 18:45:08.016704 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.016640    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77688dc558-qdt99"]
Sep 16 18:45:08.016823 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.016801    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"
Sep 16 18:45:08.023444 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.023424    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:45:08.023544 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.023509    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:45:08.023718 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.023669    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:45:08.023823 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.023718    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:45:08.023823 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.023767    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 18:45:08.036501 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.036482    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"]
Sep 16 18:45:08.036629 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.036614    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77688dc558-qdt99"
Sep 16 18:45:08.039287 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.039269    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5lf77\""
Sep 16 18:45:08.039433 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.039321    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:45:08.054461 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.054433    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"]
Sep 16 18:45:08.054535 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.054495    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"]
Sep 16 18:45:08.054535 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.054522    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vmm77"]
Sep 16 18:45:08.054647 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.054437    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.057549 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.057528    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 18:45:08.057962 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.057946    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 18:45:08.058060 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.057948    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 18:45:08.058205 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.058190    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 18:45:08.074315 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.074295    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"]
Sep 16 18:45:08.074423 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.074324    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77688dc558-qdt99"]
Sep 16 18:45:08.074423 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.074341    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vmm77"]
Sep 16 18:45:08.074536 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.074421    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:08.077070 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.077051    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dwfv6\""
Sep 16 18:45:08.077163 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.077073    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:45:08.077791 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.077770    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:45:08.092332 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.092315    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rfb8v"]
Sep 16 18:45:08.106272 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.106247    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf727ee4-c662-40c9-af1b-7050090a2851-registry-certificates\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.106367 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.106286    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf727ee4-c662-40c9-af1b-7050090a2851-trusted-ca\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.106367 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.106317    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.106485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.106370    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-bound-sa-token\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.106485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.106407    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf727ee4-c662-40c9-af1b-7050090a2851-installation-pull-secrets\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.106485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.106431    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dznz\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-kube-api-access-9dznz\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.106485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.106450    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c232a6b-df08-4275-8c74-f6b17a48b188-tmp\") pod \"klusterlet-addon-workmgr-5997c7579-gpxm9\" (UID: \"8c232a6b-df08-4275-8c74-f6b17a48b188\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"
Sep 16 18:45:08.106485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.106469    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8c232a6b-df08-4275-8c74-f6b17a48b188-klusterlet-config\") pod \"klusterlet-addon-workmgr-5997c7579-gpxm9\" (UID: \"8c232a6b-df08-4275-8c74-f6b17a48b188\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"
Sep 16 18:45:08.107013 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.106489    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf727ee4-c662-40c9-af1b-7050090a2851-image-registry-private-configuration\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.107013 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.106511    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf727ee4-c662-40c9-af1b-7050090a2851-ca-trust-extracted\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.107013 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.106550    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx45c\" (UniqueName: \"kubernetes.io/projected/8c232a6b-df08-4275-8c74-f6b17a48b188-kube-api-access-sx45c\") pod \"klusterlet-addon-workmgr-5997c7579-gpxm9\" (UID: \"8c232a6b-df08-4275-8c74-f6b17a48b188\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"
Sep 16 18:45:08.109059 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.109039    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rfb8v"]
Sep 16 18:45:08.109151 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.109137    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rfb8v"
Sep 16 18:45:08.111767 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.111626    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:45:08.111767 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.111630    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:45:08.111767 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.111730    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dgnzh\""
Sep 16 18:45:08.111972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.111800    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:45:08.155517 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.155460    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:45:08.155633 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.155460    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:45:08.155814 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.155465    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:45:08.159589 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.159561    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:08.159810 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.159790    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:08.159896 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.159876    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hrh8k\""
Sep 16 18:45:08.160017 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.160000    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:08.160092 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.160082    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:08.161389 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.161371    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pq9cp\""
Sep 16 18:45:08.207290 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207266    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27568\" (UniqueName: \"kubernetes.io/projected/8142a46b-81fa-4aab-94d4-471795cc345b-kube-api-access-27568\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.207379 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207307    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf727ee4-c662-40c9-af1b-7050090a2851-registry-certificates\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.207379 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207333    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8142a46b-81fa-4aab-94d4-471795cc345b-ca\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.207473 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207379    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8142a46b-81fa-4aab-94d4-471795cc345b-hub\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.207533 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207467    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf727ee4-c662-40c9-af1b-7050090a2851-trusted-ca\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.207533 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207510    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3733edd7-fc83-4df8-a976-80b9c7baba29-cert\") pod \"ingress-canary-rfb8v\" (UID: \"3733edd7-fc83-4df8-a976-80b9c7baba29\") " pod="openshift-ingress-canary/ingress-canary-rfb8v"
Sep 16 18:45:08.207622 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207539    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.207622 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207574    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:08.207622 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207600    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8f9b\" (UniqueName: \"kubernetes.io/projected/0432a680-1a76-4550-a03a-98c78696edae-kube-api-access-t8f9b\") pod \"managed-serviceaccount-addon-agent-77688dc558-qdt99\" (UID: \"0432a680-1a76-4550-a03a-98c78696edae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77688dc558-qdt99"
Sep 16 18:45:08.207764 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207647    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-bound-sa-token\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.207764 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207670    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e388636a-eb1f-4148-b7df-dd7bff98deed-tmp-dir\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:08.207764 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:08.207710    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:08.207764 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:08.207731    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fdb6b5c45-cv8s6: secret "image-registry-tls" not found
Sep 16 18:45:08.207764 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207711    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8142a46b-81fa-4aab-94d4-471795cc345b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.207955 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:08.207787    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls podName:bf727ee4-c662-40c9-af1b-7050090a2851 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:08.707771002 +0000 UTC m=+35.092330220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls") pod "image-registry-6fdb6b5c45-cv8s6" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851") : secret "image-registry-tls" not found
Sep 16 18:45:08.207955 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207816    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf727ee4-c662-40c9-af1b-7050090a2851-installation-pull-secrets\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.207955 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207853    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dznz\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-kube-api-access-9dznz\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.207955 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207882    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c232a6b-df08-4275-8c74-f6b17a48b188-tmp\") pod \"klusterlet-addon-workmgr-5997c7579-gpxm9\" (UID: \"8c232a6b-df08-4275-8c74-f6b17a48b188\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"
Sep 16 18:45:08.207955 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207909    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8c232a6b-df08-4275-8c74-f6b17a48b188-klusterlet-config\") pod \"klusterlet-addon-workmgr-5997c7579-gpxm9\" (UID: \"8c232a6b-df08-4275-8c74-f6b17a48b188\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"
Sep 16 18:45:08.207955 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207932    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e388636a-eb1f-4148-b7df-dd7bff98deed-config-volume\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:08.208152 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207959    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0432a680-1a76-4550-a03a-98c78696edae-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77688dc558-qdt99\" (UID: \"0432a680-1a76-4550-a03a-98c78696edae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77688dc558-qdt99"
Sep 16 18:45:08.208152 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207979    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf727ee4-c662-40c9-af1b-7050090a2851-registry-certificates\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.208152 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.207991    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf727ee4-c662-40c9-af1b-7050090a2851-image-registry-private-configuration\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.208152 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.208022    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf727ee4-c662-40c9-af1b-7050090a2851-ca-trust-extracted\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.208152 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.208071    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh4pc\" (UniqueName: \"kubernetes.io/projected/e388636a-eb1f-4148-b7df-dd7bff98deed-kube-api-access-hh4pc\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:08.208152 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.208113    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx45c\" (UniqueName: \"kubernetes.io/projected/8c232a6b-df08-4275-8c74-f6b17a48b188-kube-api-access-sx45c\") pod \"klusterlet-addon-workmgr-5997c7579-gpxm9\" (UID: \"8c232a6b-df08-4275-8c74-f6b17a48b188\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"
Sep 16 18:45:08.208152 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.208139    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8142a46b-81fa-4aab-94d4-471795cc345b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.208354 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.208185    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8142a46b-81fa-4aab-94d4-471795cc345b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.208354 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.208219    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fw6k\" (UniqueName: \"kubernetes.io/projected/3733edd7-fc83-4df8-a976-80b9c7baba29-kube-api-access-5fw6k\") pod \"ingress-canary-rfb8v\" (UID: \"3733edd7-fc83-4df8-a976-80b9c7baba29\") " pod="openshift-ingress-canary/ingress-canary-rfb8v"
Sep 16 18:45:08.208849 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.208521    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf727ee4-c662-40c9-af1b-7050090a2851-ca-trust-extracted\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.208849 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.208810    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c232a6b-df08-4275-8c74-f6b17a48b188-tmp\") pod \"klusterlet-addon-workmgr-5997c7579-gpxm9\" (UID: \"8c232a6b-df08-4275-8c74-f6b17a48b188\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"
Sep 16 18:45:08.209016 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.208950    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf727ee4-c662-40c9-af1b-7050090a2851-trusted-ca\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.212306 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.212255    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf727ee4-c662-40c9-af1b-7050090a2851-image-registry-private-configuration\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.212381 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.212312    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf727ee4-c662-40c9-af1b-7050090a2851-installation-pull-secrets\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.212442 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.212425    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8c232a6b-df08-4275-8c74-f6b17a48b188-klusterlet-config\") pod \"klusterlet-addon-workmgr-5997c7579-gpxm9\" (UID: \"8c232a6b-df08-4275-8c74-f6b17a48b188\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"
Sep 16 18:45:08.219130 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.218753    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:08.220503 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.220484    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx45c\" (UniqueName: \"kubernetes.io/projected/8c232a6b-df08-4275-8c74-f6b17a48b188-kube-api-access-sx45c\") pod \"klusterlet-addon-workmgr-5997c7579-gpxm9\" (UID: \"8c232a6b-df08-4275-8c74-f6b17a48b188\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"
Sep 16 18:45:08.221098 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.221067    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dznz\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-kube-api-access-9dznz\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.221368 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.221347    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-bound-sa-token\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.309498 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309467    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e388636a-eb1f-4148-b7df-dd7bff98deed-tmp-dir\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:08.309498 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309503    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8142a46b-81fa-4aab-94d4-471795cc345b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.309734 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309535    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e388636a-eb1f-4148-b7df-dd7bff98deed-config-volume\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:08.309734 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309551    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0432a680-1a76-4550-a03a-98c78696edae-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77688dc558-qdt99\" (UID: \"0432a680-1a76-4550-a03a-98c78696edae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77688dc558-qdt99"
Sep 16 18:45:08.309734 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309573    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh4pc\" (UniqueName: \"kubernetes.io/projected/e388636a-eb1f-4148-b7df-dd7bff98deed-kube-api-access-hh4pc\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:08.309734 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309716    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8142a46b-81fa-4aab-94d4-471795cc345b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.309942 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309747    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8142a46b-81fa-4aab-94d4-471795cc345b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.309942 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309775    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fw6k\" (UniqueName: \"kubernetes.io/projected/3733edd7-fc83-4df8-a976-80b9c7baba29-kube-api-access-5fw6k\") pod \"ingress-canary-rfb8v\" (UID: \"3733edd7-fc83-4df8-a976-80b9c7baba29\") " pod="openshift-ingress-canary/ingress-canary-rfb8v"
Sep 16 18:45:08.309942 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309807    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27568\" (UniqueName: \"kubernetes.io/projected/8142a46b-81fa-4aab-94d4-471795cc345b-kube-api-access-27568\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.309942 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309831    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8142a46b-81fa-4aab-94d4-471795cc345b-ca\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.309942 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309850    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8142a46b-81fa-4aab-94d4-471795cc345b-hub\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.309942 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309892    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3733edd7-fc83-4df8-a976-80b9c7baba29-cert\") pod \"ingress-canary-rfb8v\" (UID: \"3733edd7-fc83-4df8-a976-80b9c7baba29\") " pod="openshift-ingress-canary/ingress-canary-rfb8v"
Sep 16 18:45:08.309942 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309909    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e388636a-eb1f-4148-b7df-dd7bff98deed-tmp-dir\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:08.309942 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309931    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:08.311552 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.309963    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8f9b\" (UniqueName: \"kubernetes.io/projected/0432a680-1a76-4550-a03a-98c78696edae-kube-api-access-t8f9b\") pod \"managed-serviceaccount-addon-agent-77688dc558-qdt99\" (UID: \"0432a680-1a76-4550-a03a-98c78696edae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77688dc558-qdt99"
Sep 16 18:45:08.311552 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.310530    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8142a46b-81fa-4aab-94d4-471795cc345b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.311552 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.310576    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e388636a-eb1f-4148-b7df-dd7bff98deed-config-volume\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:08.311552 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:08.310668    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:08.311552 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:08.310751    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls podName:e388636a-eb1f-4148-b7df-dd7bff98deed nodeName:}" failed. No retries permitted until 2026-09-16 18:45:08.810732192 +0000 UTC m=+35.195291410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls") pod "dns-default-vmm77" (UID: "e388636a-eb1f-4148-b7df-dd7bff98deed") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:08.312897 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.312877    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8142a46b-81fa-4aab-94d4-471795cc345b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.312986 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.312873    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8142a46b-81fa-4aab-94d4-471795cc345b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.312986 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.312875    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0432a680-1a76-4550-a03a-98c78696edae-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-77688dc558-qdt99\" (UID: \"0432a680-1a76-4550-a03a-98c78696edae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77688dc558-qdt99"
Sep 16 18:45:08.313148 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.313047    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8142a46b-81fa-4aab-94d4-471795cc345b-hub\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.313231 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.313214    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8142a46b-81fa-4aab-94d4-471795cc345b-ca\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.313533 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.313515    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3733edd7-fc83-4df8-a976-80b9c7baba29-cert\") pod \"ingress-canary-rfb8v\" (UID: \"3733edd7-fc83-4df8-a976-80b9c7baba29\") " pod="openshift-ingress-canary/ingress-canary-rfb8v"
Sep 16 18:45:08.319125 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.319101    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh4pc\" (UniqueName: \"kubernetes.io/projected/e388636a-eb1f-4148-b7df-dd7bff98deed-kube-api-access-hh4pc\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:08.319817 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.319794    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8f9b\" (UniqueName: \"kubernetes.io/projected/0432a680-1a76-4550-a03a-98c78696edae-kube-api-access-t8f9b\") pod \"managed-serviceaccount-addon-agent-77688dc558-qdt99\" (UID: \"0432a680-1a76-4550-a03a-98c78696edae\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77688dc558-qdt99"
Sep 16 18:45:08.321374 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.321354    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fw6k\" (UniqueName: \"kubernetes.io/projected/3733edd7-fc83-4df8-a976-80b9c7baba29-kube-api-access-5fw6k\") pod \"ingress-canary-rfb8v\" (UID: \"3733edd7-fc83-4df8-a976-80b9c7baba29\") " pod="openshift-ingress-canary/ingress-canary-rfb8v"
Sep 16 18:45:08.321493 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.321477    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27568\" (UniqueName: \"kubernetes.io/projected/8142a46b-81fa-4aab-94d4-471795cc345b-kube-api-access-27568\") pod \"cluster-proxy-proxy-agent-59bf4d7768-z8zwf\" (UID: \"8142a46b-81fa-4aab-94d4-471795cc345b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.323034 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.323014    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:08.328827 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.328809    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"
Sep 16 18:45:08.351726 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.351705    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77688dc558-qdt99"
Sep 16 18:45:08.363855 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.363834    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:45:08.418283 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.418209    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rfb8v"
Sep 16 18:45:08.426151 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.426006    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:08.523268 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.523227    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:08.673755 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.673737    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77688dc558-qdt99"]
Sep 16 18:45:08.676349 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.676315    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"]
Sep 16 18:45:08.680588 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.680570    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"]
Sep 16 18:45:08.704533 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:45:08.704511    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0432a680_1a76_4550_a03a_98c78696edae.slice/crio-e1b9f7eecdb54cde543c76605b1599057b318186dd81a9d9b30b81e20645f997 WatchSource:0}: Error finding container e1b9f7eecdb54cde543c76605b1599057b318186dd81a9d9b30b81e20645f997: Status 404 returned error can't find the container with id e1b9f7eecdb54cde543c76605b1599057b318186dd81a9d9b30b81e20645f997
Sep 16 18:45:08.705114 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:45:08.705025    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c232a6b_df08_4275_8c74_f6b17a48b188.slice/crio-7797af9ca911db09481ae43a077de2092a11772ea56bc41e727ff8d22f0f732f WatchSource:0}: Error finding container 7797af9ca911db09481ae43a077de2092a11772ea56bc41e727ff8d22f0f732f: Status 404 returned error can't find the container with id 7797af9ca911db09481ae43a077de2092a11772ea56bc41e727ff8d22f0f732f
Sep 16 18:45:08.706029 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:45:08.706008    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8142a46b_81fa_4aab_94d4_471795cc345b.slice/crio-6356ba7c5edf957660a4837edf0c6cf2a00dbbf568e1c018826b3a5b97dbba9f WatchSource:0}: Error finding container 6356ba7c5edf957660a4837edf0c6cf2a00dbbf568e1c018826b3a5b97dbba9f: Status 404 returned error can't find the container with id 6356ba7c5edf957660a4837edf0c6cf2a00dbbf568e1c018826b3a5b97dbba9f
Sep 16 18:45:08.712767 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.712748    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:08.712868 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:08.712857    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:08.712907 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:08.712869    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fdb6b5c45-cv8s6: secret "image-registry-tls" not found
Sep 16 18:45:08.712907 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:08.712906    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls podName:bf727ee4-c662-40c9-af1b-7050090a2851 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:09.71289448 +0000 UTC m=+36.097453681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls") pod "image-registry-6fdb6b5c45-cv8s6" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851") : secret "image-registry-tls" not found
Sep 16 18:45:08.725705 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.725668    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rfb8v"]
Sep 16 18:45:08.727884 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:45:08.727865    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3733edd7_fc83_4df8_a976_80b9c7baba29.slice/crio-61d049f78c75d8c6a362eee45c24f8a6794582e44273069f216c12fd61533bb2 WatchSource:0}: Error finding container 61d049f78c75d8c6a362eee45c24f8a6794582e44273069f216c12fd61533bb2: Status 404 returned error can't find the container with id 61d049f78c75d8c6a362eee45c24f8a6794582e44273069f216c12fd61533bb2
Sep 16 18:45:08.813137 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.813109    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:08.813243 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:08.813227    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:08.813296 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:08.813284    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls podName:e388636a-eb1f-4148-b7df-dd7bff98deed nodeName:}" failed. No retries permitted until 2026-09-16 18:45:09.81326558 +0000 UTC m=+36.197824780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls") pod "dns-default-vmm77" (UID: "e388636a-eb1f-4148-b7df-dd7bff98deed") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:08.930923 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:08.930845    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:09.082566 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:09.082537    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:09.323256 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:09.323208    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9" event={"ID":"8c232a6b-df08-4275-8c74-f6b17a48b188","Type":"ContainerStarted","Data":"7797af9ca911db09481ae43a077de2092a11772ea56bc41e727ff8d22f0f732f"}
Sep 16 18:45:09.324583 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:09.324554    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77688dc558-qdt99" event={"ID":"0432a680-1a76-4550-a03a-98c78696edae","Type":"ContainerStarted","Data":"e1b9f7eecdb54cde543c76605b1599057b318186dd81a9d9b30b81e20645f997"}
Sep 16 18:45:09.326299 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:09.326265    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rfb8v" event={"ID":"3733edd7-fc83-4df8-a976-80b9c7baba29","Type":"ContainerStarted","Data":"61d049f78c75d8c6a362eee45c24f8a6794582e44273069f216c12fd61533bb2"}
Sep 16 18:45:09.327971 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:09.327921    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf" event={"ID":"8142a46b-81fa-4aab-94d4-471795cc345b","Type":"ContainerStarted","Data":"6356ba7c5edf957660a4837edf0c6cf2a00dbbf568e1c018826b3a5b97dbba9f"}
Sep 16 18:45:09.331603 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:09.331577    2570 generic.go:356] "Generic (PLEG): container finished" podID="d07fffcd-979c-415a-83be-27c984957d2a" containerID="3dd1235da0ef690d887863af70c039df0763c1b04d190dcffb96d22b3041256f" exitCode=0
Sep 16 18:45:09.331725 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:09.331612    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbd6r" event={"ID":"d07fffcd-979c-415a-83be-27c984957d2a","Type":"ContainerDied","Data":"3dd1235da0ef690d887863af70c039df0763c1b04d190dcffb96d22b3041256f"}
Sep 16 18:45:09.461600 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:09.461559    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:09.721279 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:09.720531    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:09.721279 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:09.720759    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:09.721279 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:09.720779    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fdb6b5c45-cv8s6: secret "image-registry-tls" not found
Sep 16 18:45:09.721279 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:09.720837    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls podName:bf727ee4-c662-40c9-af1b-7050090a2851 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:11.720819474 +0000 UTC m=+38.105378695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls") pod "image-registry-6fdb6b5c45-cv8s6" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851") : secret "image-registry-tls" not found
Sep 16 18:45:09.822317 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:09.821708    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:09.822317 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:09.821914    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:09.822317 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:09.821986    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls podName:e388636a-eb1f-4148-b7df-dd7bff98deed nodeName:}" failed. No retries permitted until 2026-09-16 18:45:11.821967837 +0000 UTC m=+38.206527057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls") pod "dns-default-vmm77" (UID: "e388636a-eb1f-4148-b7df-dd7bff98deed") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:10.344507 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:10.344472    2570 generic.go:356] "Generic (PLEG): container finished" podID="d07fffcd-979c-415a-83be-27c984957d2a" containerID="dde7c6c10b861577ae0367c3ac5b23816599ba2ae2d9a7c0de2e10a8d52d21eb" exitCode=0
Sep 16 18:45:10.344663 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:10.344537    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbd6r" event={"ID":"d07fffcd-979c-415a-83be-27c984957d2a","Type":"ContainerDied","Data":"dde7c6c10b861577ae0367c3ac5b23816599ba2ae2d9a7c0de2e10a8d52d21eb"}
Sep 16 18:45:10.452063 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:10.451096    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:10.539919 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:10.539867    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:11.352641 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:11.352558    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pbd6r" event={"ID":"d07fffcd-979c-415a-83be-27c984957d2a","Type":"ContainerStarted","Data":"1171eec0c9e9a32436627fb25c7220b7d55cd42977cba77408ac7a57df659954"}
Sep 16 18:45:11.375379 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:11.375289    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pbd6r" podStartSLOduration=8.05633253 podStartE2EDuration="36.375268458s" totalImagesPullingTime="28.318935928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:11.374744313 +0000 UTC m=+37.759303538" watchObservedRunningTime="2026-09-16 18:45:11.375268458 +0000 UTC m=+37.759827681"
Sep 16 18:45:11.499574 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:11.499538    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:11.598991 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:11.598939    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:11.738386 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:11.738229    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:11.738386 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:11.738377    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:11.738638 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:11.738394    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fdb6b5c45-cv8s6: secret "image-registry-tls" not found
Sep 16 18:45:11.738638 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:11.738452    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls podName:bf727ee4-c662-40c9-af1b-7050090a2851 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.738433686 +0000 UTC m=+42.122992902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls") pod "image-registry-6fdb6b5c45-cv8s6" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851") : secret "image-registry-tls" not found
Sep 16 18:45:11.838916 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:11.838877    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:11.839085 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:11.839046    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:11.839147 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:11.839124    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls podName:e388636a-eb1f-4148-b7df-dd7bff98deed nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.839103041 +0000 UTC m=+42.223662259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls") pod "dns-default-vmm77" (UID: "e388636a-eb1f-4148-b7df-dd7bff98deed") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:13.309594 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:13.309558    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:13.385109 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:13.385068    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:15.764601 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:15.764556    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:15.764996 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:15.764745    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:15.764996 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:15.764768    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fdb6b5c45-cv8s6: secret "image-registry-tls" not found
Sep 16 18:45:15.764996 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:15.764840    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls podName:bf727ee4-c662-40c9-af1b-7050090a2851 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:23.764823432 +0000 UTC m=+50.149382633 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls") pod "image-registry-6fdb6b5c45-cv8s6" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851") : secret "image-registry-tls" not found
Sep 16 18:45:15.865425 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:15.865392    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:15.865574 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:15.865550    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:15.865649 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:15.865636    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls podName:e388636a-eb1f-4148-b7df-dd7bff98deed nodeName:}" failed. No retries permitted until 2026-09-16 18:45:23.865615296 +0000 UTC m=+50.250174498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls") pod "dns-default-vmm77" (UID: "e388636a-eb1f-4148-b7df-dd7bff98deed") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:18.371463 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:18.371422    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rfb8v" event={"ID":"3733edd7-fc83-4df8-a976-80b9c7baba29","Type":"ContainerStarted","Data":"d4c9b8470e0afc4824cc7883ae17862aa68e402bf7b198c368b9817462002236"}
Sep 16 18:45:18.373523 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:18.373496    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf" event={"ID":"8142a46b-81fa-4aab-94d4-471795cc345b","Type":"ContainerStarted","Data":"6c3888a9c392623144d55fd5c19ad95d3d941da0a373f77d8573a8f41ef9c8f0"}
Sep 16 18:45:18.373638 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:18.373533    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf" event={"ID":"8142a46b-81fa-4aab-94d4-471795cc345b","Type":"ContainerStarted","Data":"d4adcd27603c82547e49533c7d4ded0744e9a8d0cee82ef3e3cb320b8ac3d879"}
Sep 16 18:45:18.373638 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:18.373547    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf" event={"ID":"8142a46b-81fa-4aab-94d4-471795cc345b","Type":"ContainerStarted","Data":"4179a1f8ddf9b6c4b5e9dfbe212d9b109e7dda26c0f46e97225d043bc0d9d6a0"}
Sep 16 18:45:18.374917 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:18.374893    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9" event={"ID":"8c232a6b-df08-4275-8c74-f6b17a48b188","Type":"ContainerStarted","Data":"e6cfe00579dc806dfb39d2fa59e5e659878861bbbe4c47801b5ab88a72597ffe"}
Sep 16 18:45:18.375074 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:18.375039    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"
Sep 16 18:45:18.376949 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:18.376907    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9"
Sep 16 18:45:18.378240 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:18.378214    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77688dc558-qdt99" event={"ID":"0432a680-1a76-4550-a03a-98c78696edae","Type":"ContainerStarted","Data":"c338fbae7008eba465f2ab9a2788d58b2b28b9795588a72e0b7ee867f462e236"}
Sep 16 18:45:18.413623 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:18.413583    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5997c7579-gpxm9" podStartSLOduration=19.792423957 podStartE2EDuration="28.413571088s" totalImagesPullingTime="8.621147131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:18.412935521 +0000 UTC m=+44.797494756" watchObservedRunningTime="2026-09-16 18:45:18.413571088 +0000 UTC m=+44.798130311"
Sep 16 18:45:18.413896 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:18.413873    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rfb8v" podStartSLOduration=1.792369126 podStartE2EDuration="10.413867976s" totalImagesPullingTime="8.62149885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:18.396713742 +0000 UTC m=+44.781272970" watchObservedRunningTime="2026-09-16 18:45:18.413867976 +0000 UTC m=+44.798427199"
Sep 16 18:45:18.463483 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:18.463441    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-77688dc558-qdt99" podStartSLOduration=20.83437308 podStartE2EDuration="29.463423887s" totalImagesPullingTime="8.629050807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:18.436860605 +0000 UTC m=+44.821419828" watchObservedRunningTime="2026-09-16 18:45:18.463423887 +0000 UTC m=+44.847983112"
Sep 16 18:45:18.463587 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:18.463514    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf" podStartSLOduration=19.842358902 podStartE2EDuration="28.463504053s" totalImagesPullingTime="8.621145151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:18.463115367 +0000 UTC m=+44.847674589" watchObservedRunningTime="2026-09-16 18:45:18.463504053 +0000 UTC m=+44.848063277"
Sep 16 18:45:22.533202 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:22.533168    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:23.820704 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:23.820658    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:23.821053 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:23.820812    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:23.821053 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:23.820832    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fdb6b5c45-cv8s6: secret "image-registry-tls" not found
Sep 16 18:45:23.821053 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:23.820894    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls podName:bf727ee4-c662-40c9-af1b-7050090a2851 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:39.820879312 +0000 UTC m=+66.205438517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls") pod "image-registry-6fdb6b5c45-cv8s6" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851") : secret "image-registry-tls" not found
Sep 16 18:45:23.921885 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:23.921856    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:23.922025 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:23.922000    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:23.922069 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:23.922057    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls podName:e388636a-eb1f-4148-b7df-dd7bff98deed nodeName:}" failed. No retries permitted until 2026-09-16 18:45:39.922041016 +0000 UTC m=+66.306600232 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls") pod "dns-default-vmm77" (UID: "e388636a-eb1f-4148-b7df-dd7bff98deed") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:23.923076 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:23.923044    2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:33.314768 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:33.314735    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rbsh"
Sep 16 18:45:39.728077 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:39.728043    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:45:39.731029 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:39.731012    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:39.741733 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:39.741711    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2e821ae-3c87-4fa1-8065-4004d791b3fc-original-pull-secret\") pod \"global-pull-secret-syncer-49tcg\" (UID: \"b2e821ae-3c87-4fa1-8065-4004d791b3fc\") " pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:45:39.828981 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:39.828957    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:45:39.829072 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:39.829002    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs\") pod \"network-metrics-daemon-bwr7v\" (UID: \"24c2659a-61bd-4dca-aba0-d56072325bde\") " pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:45:39.829129 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:39.829084    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:39.829129 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:39.829102    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fdb6b5c45-cv8s6: secret "image-registry-tls" not found
Sep 16 18:45:39.829223 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:39.829144    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls podName:bf727ee4-c662-40c9-af1b-7050090a2851 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:11.829131185 +0000 UTC m=+98.213690386 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls") pod "image-registry-6fdb6b5c45-cv8s6" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851") : secret "image-registry-tls" not found
Sep 16 18:45:39.831557 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:39.831543    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:39.839765 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:39.839752    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:45:39.839836 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:39.839793    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs podName:24c2659a-61bd-4dca-aba0-d56072325bde nodeName:}" failed. No retries permitted until 2026-09-16 18:46:43.839781848 +0000 UTC m=+130.224341048 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs") pod "network-metrics-daemon-bwr7v" (UID: "24c2659a-61bd-4dca-aba0-d56072325bde") : secret "metrics-daemon-secret" not found
Sep 16 18:45:39.929866 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:39.929845    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:45:39.929934 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:39.929875    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm624\" (UniqueName: \"kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624\") pod \"network-check-target-47c96\" (UID: \"42e26367-9854-47ff-8046-3cd71c2b851a\") " pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:45:39.930013 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:39.929994    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:39.930072 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:45:39.930056    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls podName:e388636a-eb1f-4148-b7df-dd7bff98deed nodeName:}" failed. No retries permitted until 2026-09-16 18:46:11.930040402 +0000 UTC m=+98.314599606 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls") pod "dns-default-vmm77" (UID: "e388636a-eb1f-4148-b7df-dd7bff98deed") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:39.932574 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:39.932557    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:39.942905 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:39.942884    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:39.953134 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:39.953108    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm624\" (UniqueName: \"kubernetes.io/projected/42e26367-9854-47ff-8046-3cd71c2b851a-kube-api-access-bm624\") pod \"network-check-target-47c96\" (UID: \"42e26367-9854-47ff-8046-3cd71c2b851a\") " pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:45:39.968721 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:39.968677    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hrh8k\""
Sep 16 18:45:39.972609 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:39.972595    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-49tcg"
Sep 16 18:45:39.976391 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:39.976377    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:45:40.243142 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:40.243076    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-49tcg"]
Sep 16 18:45:40.246867 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:40.245785    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-47c96"]
Sep 16 18:45:40.248440 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:45:40.248380    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e821ae_3c87_4fa1_8065_4004d791b3fc.slice/crio-770a25e68569fcbc4ffcb709040d792882518a2d8eb4cddfcf9d66a5a0811604 WatchSource:0}: Error finding container 770a25e68569fcbc4ffcb709040d792882518a2d8eb4cddfcf9d66a5a0811604: Status 404 returned error can't find the container with id 770a25e68569fcbc4ffcb709040d792882518a2d8eb4cddfcf9d66a5a0811604
Sep 16 18:45:40.250469 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:45:40.250424    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e26367_9854_47ff_8046_3cd71c2b851a.slice/crio-39df1361f1c8448d047ffc1a2c4f74d3fd129b43fe58fc53094321a989ef36a0 WatchSource:0}: Error finding container 39df1361f1c8448d047ffc1a2c4f74d3fd129b43fe58fc53094321a989ef36a0: Status 404 returned error can't find the container with id 39df1361f1c8448d047ffc1a2c4f74d3fd129b43fe58fc53094321a989ef36a0
Sep 16 18:45:40.444136 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:40.444104    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-47c96" event={"ID":"42e26367-9854-47ff-8046-3cd71c2b851a","Type":"ContainerStarted","Data":"39df1361f1c8448d047ffc1a2c4f74d3fd129b43fe58fc53094321a989ef36a0"}
Sep 16 18:45:40.445173 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:40.445118    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-49tcg" event={"ID":"b2e821ae-3c87-4fa1-8065-4004d791b3fc","Type":"ContainerStarted","Data":"770a25e68569fcbc4ffcb709040d792882518a2d8eb4cddfcf9d66a5a0811604"}
Sep 16 18:45:45.467766 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:45.467645    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-49tcg" event={"ID":"b2e821ae-3c87-4fa1-8065-4004d791b3fc","Type":"ContainerStarted","Data":"b4a8a657b54534066e9c8badcc1f4769584e7a29c000b846b20a871eeb04fb20"}
Sep 16 18:45:45.469255 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:45.469228    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-47c96" event={"ID":"42e26367-9854-47ff-8046-3cd71c2b851a","Type":"ContainerStarted","Data":"9acde502a02ec61644a04bdabefd629ec614273aa0fc2530b5a5c01281f87339"}
Sep 16 18:45:45.469368 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:45.469352    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:45:45.482183 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:45.482133    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-49tcg" podStartSLOduration=65.616065851 podStartE2EDuration="1m10.482117902s" totalImagesPullingTime="4.866052051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:45.481930369 +0000 UTC m=+71.866489603" watchObservedRunningTime="2026-09-16 18:45:45.482117902 +0000 UTC m=+71.866677127"
Sep 16 18:45:45.497406 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:45:45.497371    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-47c96" podStartSLOduration=65.643267873 podStartE2EDuration="1m10.497357553s" totalImagesPullingTime="4.85408968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:45.497070586 +0000 UTC m=+71.881629820" watchObservedRunningTime="2026-09-16 18:45:45.497357553 +0000 UTC m=+71.881916776"
Sep 16 18:46:11.843869 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:11.843828    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls\") pod \"image-registry-6fdb6b5c45-cv8s6\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") " pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:46:11.844234 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:11.843988    2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:11.844234 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:11.844005    2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fdb6b5c45-cv8s6: secret "image-registry-tls" not found
Sep 16 18:46:11.844234 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:11.844082    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls podName:bf727ee4-c662-40c9-af1b-7050090a2851 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:15.844066717 +0000 UTC m=+162.228625918 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls") pod "image-registry-6fdb6b5c45-cv8s6" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851") : secret "image-registry-tls" not found
Sep 16 18:46:11.944933 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:11.944910    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:46:11.945040 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:11.945028    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:11.945085 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:11.945072    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls podName:e388636a-eb1f-4148-b7df-dd7bff98deed nodeName:}" failed. No retries permitted until 2026-09-16 18:47:15.945060787 +0000 UTC m=+162.329619988 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls") pod "dns-default-vmm77" (UID: "e388636a-eb1f-4148-b7df-dd7bff98deed") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:12.377812 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.377779    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d9v42"]
Sep 16 18:46:12.380057 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.380036    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:12.382363 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.382343    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:46:12.382467 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.382377    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hz5t4\""
Sep 16 18:46:12.383249 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.383231    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:46:12.383334 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.383236    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:46:12.383334 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.383266    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:46:12.395175 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.395155    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d9v42"]
Sep 16 18:46:12.448234 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.448211    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:12.448328 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.448243    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-data-volume\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:12.448328 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.448276    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:12.448430 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.448351    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-crio-socket\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:12.448488 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.448450    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chc22\" (UniqueName: \"kubernetes.io/projected/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-api-access-chc22\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:12.548911 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.548889    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-data-volume\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:12.549005 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.548931    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:12.549005 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.548961    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-crio-socket\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:12.549116 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.549013    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-crio-socket\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:12.549116 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.549042    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chc22\" (UniqueName: \"kubernetes.io/projected/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-api-access-chc22\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:12.549116 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.549061    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:12.549116 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:12.549065    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:12.549116 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:12.549110    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:12.549301 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:12.549137    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls podName:78a486cb-98bc-47ab-9679-c1bc9e3fc69d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.049117468 +0000 UTC m=+99.433676684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d9v42" (UID: "78a486cb-98bc-47ab-9679-c1bc9e3fc69d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:12.549301 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:12.549162    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm podName:78a486cb-98bc-47ab-9679-c1bc9e3fc69d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.049146629 +0000 UTC m=+99.433705845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d9v42" (UID: "78a486cb-98bc-47ab-9679-c1bc9e3fc69d") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:12.549301 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.549218    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-data-volume\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:12.559444 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:12.559420    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chc22\" (UniqueName: \"kubernetes.io/projected/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-api-access-chc22\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:13.051817 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:13.051786    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:13.052179 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:13.051841    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:13.052179 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:13.051912    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:13.052179 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:13.051956    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:13.052179 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:13.051993    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm podName:78a486cb-98bc-47ab-9679-c1bc9e3fc69d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:14.051973273 +0000 UTC m=+100.436532488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d9v42" (UID: "78a486cb-98bc-47ab-9679-c1bc9e3fc69d") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:13.052179 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:13.052009    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls podName:78a486cb-98bc-47ab-9679-c1bc9e3fc69d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:14.052001237 +0000 UTC m=+100.436560438 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d9v42" (UID: "78a486cb-98bc-47ab-9679-c1bc9e3fc69d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:14.057790 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:14.057761    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:14.058196 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:14.057810    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:14.058196 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:14.057878    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:14.058196 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:14.057903    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:14.058196 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:14.057940    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm podName:78a486cb-98bc-47ab-9679-c1bc9e3fc69d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:16.057926109 +0000 UTC m=+102.442485336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d9v42" (UID: "78a486cb-98bc-47ab-9679-c1bc9e3fc69d") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:14.058196 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:14.057956    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls podName:78a486cb-98bc-47ab-9679-c1bc9e3fc69d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:16.057948215 +0000 UTC m=+102.442507416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d9v42" (UID: "78a486cb-98bc-47ab-9679-c1bc9e3fc69d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:15.328454 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:15.328426    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jsj4b_41b9e54d-41ed-4362-a99a-9998cba62a60/dns-node-resolver/0.log"
Sep 16 18:46:16.073196 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:16.073167    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:16.073343 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:16.073215    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:16.073343 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:16.073281    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:16.073343 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:16.073299    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:16.073343 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:16.073344    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm podName:78a486cb-98bc-47ab-9679-c1bc9e3fc69d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:20.07333008 +0000 UTC m=+106.457889311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d9v42" (UID: "78a486cb-98bc-47ab-9679-c1bc9e3fc69d") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:16.073482 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:16.073359    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls podName:78a486cb-98bc-47ab-9679-c1bc9e3fc69d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:20.073353209 +0000 UTC m=+106.457912447 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d9v42" (UID: "78a486cb-98bc-47ab-9679-c1bc9e3fc69d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:16.130621 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:16.130595    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mzfr8_41f91dbd-52af-482f-88fe-b09f0e5c48cc/node-ca/0.log"
Sep 16 18:46:16.475105 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:16.475040    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-47c96"
Sep 16 18:46:17.130841 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:17.130812    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rfb8v_3733edd7-fc83-4df8-a976-80b9c7baba29/serve-healthcheck-canary/0.log"
Sep 16 18:46:20.104153 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:20.104116    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:20.104614 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:20.104175    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:20.104614 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:20.104238    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:20.104614 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:20.104258    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:20.104614 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:20.104300    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm podName:78a486cb-98bc-47ab-9679-c1bc9e3fc69d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:28.104286477 +0000 UTC m=+114.488845678 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d9v42" (UID: "78a486cb-98bc-47ab-9679-c1bc9e3fc69d") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:20.104614 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:20.104314    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls podName:78a486cb-98bc-47ab-9679-c1bc9e3fc69d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:28.104308263 +0000 UTC m=+114.488867463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d9v42" (UID: "78a486cb-98bc-47ab-9679-c1bc9e3fc69d") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:28.163050 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:28.163019    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:28.163465 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:28.163068    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:28.163506 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:28.163484    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:28.165226 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:28.165206    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78a486cb-98bc-47ab-9679-c1bc9e3fc69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9v42\" (UID: \"78a486cb-98bc-47ab-9679-c1bc9e3fc69d\") " pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:28.288931 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:28.288907    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d9v42"
Sep 16 18:46:28.580477 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:28.580451    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d9v42"]
Sep 16 18:46:28.582752 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:46:28.582728    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a486cb_98bc_47ab_9679_c1bc9e3fc69d.slice/crio-59cdd663c1019239dee7defcbd2771d6ac71bf9ccbef983692bf2780601567a6 WatchSource:0}: Error finding container 59cdd663c1019239dee7defcbd2771d6ac71bf9ccbef983692bf2780601567a6: Status 404 returned error can't find the container with id 59cdd663c1019239dee7defcbd2771d6ac71bf9ccbef983692bf2780601567a6
Sep 16 18:46:28.596636 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:28.596601    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d9v42" event={"ID":"78a486cb-98bc-47ab-9679-c1bc9e3fc69d","Type":"ContainerStarted","Data":"59cdd663c1019239dee7defcbd2771d6ac71bf9ccbef983692bf2780601567a6"}
Sep 16 18:46:29.601737 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:29.601704    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d9v42" event={"ID":"78a486cb-98bc-47ab-9679-c1bc9e3fc69d","Type":"ContainerStarted","Data":"c04f21bb3b693bfc83e889e4d3c5beee5cd73358d9cd0e221033ac6b492207b0"}
Sep 16 18:46:29.602086 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:29.601757    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d9v42" event={"ID":"78a486cb-98bc-47ab-9679-c1bc9e3fc69d","Type":"ContainerStarted","Data":"0bebdeb1e16c05fb48cb41ec27e7b17a82d7e910216c30d4a49ca3a53e92a1ed"}
Sep 16 18:46:30.606321 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:30.606287    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d9v42" event={"ID":"78a486cb-98bc-47ab-9679-c1bc9e3fc69d","Type":"ContainerStarted","Data":"48b9a6b111091eee69ea838a24fe0f7b854a253aa31bb7cc384bc1b1be4239e0"}
Sep 16 18:46:30.628202 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:30.628162    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d9v42" podStartSLOduration=16.875956274 podStartE2EDuration="18.628146865s" totalImagesPullingTime="1.752190591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:30.628082802 +0000 UTC m=+117.012642026" watchObservedRunningTime="2026-09-16 18:46:30.628146865 +0000 UTC m=+117.012706120"
Sep 16 18:46:38.365065 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:38.365021    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf" podUID="8142a46b-81fa-4aab-94d4-471795cc345b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:46:43.875555 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:43.875523    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs\") pod \"network-metrics-daemon-bwr7v\" (UID: \"24c2659a-61bd-4dca-aba0-d56072325bde\") " pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:46:43.877776 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:43.877754    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c2659a-61bd-4dca-aba0-d56072325bde-metrics-certs\") pod \"network-metrics-daemon-bwr7v\" (UID: \"24c2659a-61bd-4dca-aba0-d56072325bde\") " pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:46:43.883062 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:43.883040    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pq9cp\""
Sep 16 18:46:43.890171 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:43.890154    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bwr7v"
Sep 16 18:46:43.929852 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:43.929823    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-md657"]
Sep 16 18:46:43.932541 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:43.932525    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:43.936274 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:43.936251    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:46:43.936371 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:43.936257    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:46:43.936371 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:43.936316    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:46:43.937679 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:43.937519    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:46:43.938152 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:43.938132    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:46:43.938254 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:43.938209    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gf67p\""
Sep 16 18:46:43.938321 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:43.938264    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:46:44.077156 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.077125    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-wtmp\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.077336 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.077179    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-sys\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.077336 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.077231    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.077336 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.077298    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-textfile\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.077336 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.077332    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.077506 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.077366    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9pq\" (UniqueName: \"kubernetes.io/projected/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-kube-api-access-xf9pq\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.077506 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.077394    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-tls\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.077506 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.077453    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-root\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.077644 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.077520    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-metrics-client-ca\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.162814 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.162718    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bwr7v"]
Sep 16 18:46:44.165347 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:46:44.165317    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c2659a_61bd_4dca_aba0_d56072325bde.slice/crio-a546a44e32fbb7c89c2966ad93e29ee0e36b36953a7b7d13a2c130ff93cb1a6f WatchSource:0}: Error finding container a546a44e32fbb7c89c2966ad93e29ee0e36b36953a7b7d13a2c130ff93cb1a6f: Status 404 returned error can't find the container with id a546a44e32fbb7c89c2966ad93e29ee0e36b36953a7b7d13a2c130ff93cb1a6f
Sep 16 18:46:44.178821 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.178798    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-wtmp\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.178907 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.178845    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-sys\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.178907 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.178866    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.178907 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.178885    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-textfile\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.178907 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.178901    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.179060 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.178926    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9pq\" (UniqueName: \"kubernetes.io/projected/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-kube-api-access-xf9pq\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.179060 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.178942    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-tls\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.179060 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.178956    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-root\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.179060 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.178957    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-wtmp\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.179060 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.178994    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-sys\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.179060 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.179015    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-root\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.179060 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.179024    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-metrics-client-ca\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.179375 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.179269    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-textfile\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.179498 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.179477    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-accelerators-collector-config\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.179498 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.179492    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-metrics-client-ca\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.181221 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.181204    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-tls\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.181292 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.181231    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.188272 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.188245    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9pq\" (UniqueName: \"kubernetes.io/projected/9a9cce6a-0af1-4da9-abd9-a55db5f340e6-kube-api-access-xf9pq\") pod \"node-exporter-md657\" (UID: \"9a9cce6a-0af1-4da9-abd9-a55db5f340e6\") " pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.244095 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.244071    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-md657"
Sep 16 18:46:44.253464 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:46:44.253442    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9cce6a_0af1_4da9_abd9_a55db5f340e6.slice/crio-c7a3d435238a0cba2488c28e82686e72b0dd685c6dd284deef263055f92e204a WatchSource:0}: Error finding container c7a3d435238a0cba2488c28e82686e72b0dd685c6dd284deef263055f92e204a: Status 404 returned error can't find the container with id c7a3d435238a0cba2488c28e82686e72b0dd685c6dd284deef263055f92e204a
Sep 16 18:46:44.648013 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.647972    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bwr7v" event={"ID":"24c2659a-61bd-4dca-aba0-d56072325bde","Type":"ContainerStarted","Data":"a546a44e32fbb7c89c2966ad93e29ee0e36b36953a7b7d13a2c130ff93cb1a6f"}
Sep 16 18:46:44.649206 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:44.649173    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-md657" event={"ID":"9a9cce6a-0af1-4da9-abd9-a55db5f340e6","Type":"ContainerStarted","Data":"c7a3d435238a0cba2488c28e82686e72b0dd685c6dd284deef263055f92e204a"}
Sep 16 18:46:45.656613 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:45.656579    2570 generic.go:356] "Generic (PLEG): container finished" podID="9a9cce6a-0af1-4da9-abd9-a55db5f340e6" containerID="5e87d6adfdfd2076fbdc498caed72ef4b1f7b198897a9e3be9c3fe64de6699c4" exitCode=0
Sep 16 18:46:45.656972 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:45.656664    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-md657" event={"ID":"9a9cce6a-0af1-4da9-abd9-a55db5f340e6","Type":"ContainerDied","Data":"5e87d6adfdfd2076fbdc498caed72ef4b1f7b198897a9e3be9c3fe64de6699c4"}
Sep 16 18:46:45.658488 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:45.658465    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bwr7v" event={"ID":"24c2659a-61bd-4dca-aba0-d56072325bde","Type":"ContainerStarted","Data":"6cfe26ff6ce38aaa2c32d35f5d5cc1d92cee2f4c6f0ffe019178babd940f6e91"}
Sep 16 18:46:45.658596 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:45.658510    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bwr7v" event={"ID":"24c2659a-61bd-4dca-aba0-d56072325bde","Type":"ContainerStarted","Data":"4f1c75216eebd1bfb6255cd8715baeb7fae52b8402013ca3999c3997ee6a4e9f"}
Sep 16 18:46:45.692679 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:45.692635    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bwr7v" podStartSLOduration=129.904521869 podStartE2EDuration="2m10.692621587s" totalImagesPullingTime="788.099718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:45.691413497 +0000 UTC m=+132.075972721" watchObservedRunningTime="2026-09-16 18:46:45.692621587 +0000 UTC m=+132.077180809"
Sep 16 18:46:46.663821 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:46.663781    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-md657" event={"ID":"9a9cce6a-0af1-4da9-abd9-a55db5f340e6","Type":"ContainerStarted","Data":"43b9fb07e84c081913744bc9a278b248e5b4bd97553e16e4c213831ce494b152"}
Sep 16 18:46:46.664221 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:46.663837    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-md657" event={"ID":"9a9cce6a-0af1-4da9-abd9-a55db5f340e6","Type":"ContainerStarted","Data":"fc1b1337bbc7093de50ae10f13aec0697794d54bc835223aa578614daf73a916"}
Sep 16 18:46:48.365216 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:48.365179    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf" podUID="8142a46b-81fa-4aab-94d4-471795cc345b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:46:57.200815 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.200762    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-md657" podStartSLOduration=12.980383923 podStartE2EDuration="14.20074513s" totalImagesPullingTime="1.15982318s" totalInitContainerRuntime="60.538027ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:46.685505774 +0000 UTC m=+133.070064997" watchObservedRunningTime="2026-09-16 18:46:57.20074513 +0000 UTC m=+143.585304352"
Sep 16 18:46:57.201372 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.201356    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"]
Sep 16 18:46:57.201522 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:46:57.201507    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6" podUID="bf727ee4-c662-40c9-af1b-7050090a2851"
Sep 16 18:46:57.696444 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.696407    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:46:57.701753 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.701729    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:46:57.876389 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.876356    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-bound-sa-token\") pod \"bf727ee4-c662-40c9-af1b-7050090a2851\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") "
Sep 16 18:46:57.876389 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.876391    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dznz\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-kube-api-access-9dznz\") pod \"bf727ee4-c662-40c9-af1b-7050090a2851\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") "
Sep 16 18:46:57.876602 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.876414    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf727ee4-c662-40c9-af1b-7050090a2851-registry-certificates\") pod \"bf727ee4-c662-40c9-af1b-7050090a2851\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") "
Sep 16 18:46:57.876602 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.876448    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf727ee4-c662-40c9-af1b-7050090a2851-installation-pull-secrets\") pod \"bf727ee4-c662-40c9-af1b-7050090a2851\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") "
Sep 16 18:46:57.876602 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.876486    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf727ee4-c662-40c9-af1b-7050090a2851-ca-trust-extracted\") pod \"bf727ee4-c662-40c9-af1b-7050090a2851\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") "
Sep 16 18:46:57.876602 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.876511    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf727ee4-c662-40c9-af1b-7050090a2851-image-registry-private-configuration\") pod \"bf727ee4-c662-40c9-af1b-7050090a2851\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") "
Sep 16 18:46:57.876602 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.876542    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf727ee4-c662-40c9-af1b-7050090a2851-trusted-ca\") pod \"bf727ee4-c662-40c9-af1b-7050090a2851\" (UID: \"bf727ee4-c662-40c9-af1b-7050090a2851\") "
Sep 16 18:46:57.876901 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.876780    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf727ee4-c662-40c9-af1b-7050090a2851-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bf727ee4-c662-40c9-af1b-7050090a2851" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:57.876901 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.876793    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf727ee4-c662-40c9-af1b-7050090a2851-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bf727ee4-c662-40c9-af1b-7050090a2851" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:57.877040 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.877018    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf727ee4-c662-40c9-af1b-7050090a2851-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bf727ee4-c662-40c9-af1b-7050090a2851" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:57.878890 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.878861    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bf727ee4-c662-40c9-af1b-7050090a2851" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:57.879080 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.879056    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-kube-api-access-9dznz" (OuterVolumeSpecName: "kube-api-access-9dznz") pod "bf727ee4-c662-40c9-af1b-7050090a2851" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851"). InnerVolumeSpecName "kube-api-access-9dznz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:57.879221 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.879198    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf727ee4-c662-40c9-af1b-7050090a2851-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bf727ee4-c662-40c9-af1b-7050090a2851" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:57.879307 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.879283    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf727ee4-c662-40c9-af1b-7050090a2851-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bf727ee4-c662-40c9-af1b-7050090a2851" (UID: "bf727ee4-c662-40c9-af1b-7050090a2851"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:57.977984 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.977929    2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bf727ee4-c662-40c9-af1b-7050090a2851-ca-trust-extracted\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:46:57.977984 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.977951    2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bf727ee4-c662-40c9-af1b-7050090a2851-image-registry-private-configuration\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:46:57.977984 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.977961    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bf727ee4-c662-40c9-af1b-7050090a2851-trusted-ca\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:46:57.977984 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.977970    2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-bound-sa-token\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:46:57.977984 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.977978    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dznz\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-kube-api-access-9dznz\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:46:57.977984 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.977986    2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bf727ee4-c662-40c9-af1b-7050090a2851-registry-certificates\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:46:57.978223 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:57.977995    2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bf727ee4-c662-40c9-af1b-7050090a2851-installation-pull-secrets\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:46:58.364754 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:58.364718    2570 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf" podUID="8142a46b-81fa-4aab-94d4-471795cc345b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:46:58.365083 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:58.364782    2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf"
Sep 16 18:46:58.365239 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:58.365212    2570 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6c3888a9c392623144d55fd5c19ad95d3d941da0a373f77d8573a8f41ef9c8f0"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 16 18:46:58.365275 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:58.365262    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf" podUID="8142a46b-81fa-4aab-94d4-471795cc345b" containerName="service-proxy" containerID="cri-o://6c3888a9c392623144d55fd5c19ad95d3d941da0a373f77d8573a8f41ef9c8f0" gracePeriod=30
Sep 16 18:46:58.701936 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:58.701863    2570 generic.go:356] "Generic (PLEG): container finished" podID="8142a46b-81fa-4aab-94d4-471795cc345b" containerID="6c3888a9c392623144d55fd5c19ad95d3d941da0a373f77d8573a8f41ef9c8f0" exitCode=2
Sep 16 18:46:58.702083 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:58.701943    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf" event={"ID":"8142a46b-81fa-4aab-94d4-471795cc345b","Type":"ContainerDied","Data":"6c3888a9c392623144d55fd5c19ad95d3d941da0a373f77d8573a8f41ef9c8f0"}
Sep 16 18:46:58.702083 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:58.701993    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59bf4d7768-z8zwf" event={"ID":"8142a46b-81fa-4aab-94d4-471795cc345b","Type":"ContainerStarted","Data":"0b88c618f4921b5eea1a3118c597a61f4dd0822386162c0c82949f7d9c2ec879"}
Sep 16 18:46:58.702083 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:58.701959    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"
Sep 16 18:46:58.735339 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:58.735314    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"]
Sep 16 18:46:58.743296 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:58.743275    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6fdb6b5c45-cv8s6"]
Sep 16 18:46:58.884919 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:46:58.884891    2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bf727ee4-c662-40c9-af1b-7050090a2851-registry-tls\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:47:00.159601 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:47:00.159565    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf727ee4-c662-40c9-af1b-7050090a2851" path="/var/lib/kubelet/pods/bf727ee4-c662-40c9-af1b-7050090a2851/volumes"
Sep 16 18:47:11.101698 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:47:11.101640    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vmm77" podUID="e388636a-eb1f-4148-b7df-dd7bff98deed"
Sep 16 18:47:15.999166 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:47:15.999132    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:47:16.001289 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:47:16.001272    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e388636a-eb1f-4148-b7df-dd7bff98deed-metrics-tls\") pod \"dns-default-vmm77\" (UID: \"e388636a-eb1f-4148-b7df-dd7bff98deed\") " pod="openshift-dns/dns-default-vmm77"
Sep 16 18:47:23.155817 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:47:23.155782    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vmm77"
Sep 16 18:47:23.158695 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:47:23.158666    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dwfv6\""
Sep 16 18:47:23.166896 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:47:23.166879    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vmm77"
Sep 16 18:47:23.439666 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:47:23.439601    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vmm77"]
Sep 16 18:47:23.443027 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:47:23.442995    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode388636a_eb1f_4148_b7df_dd7bff98deed.slice/crio-7e040a511aaa89980310a578a6eccbc68a390b0767406a98c2ed863c2017978c WatchSource:0}: Error finding container 7e040a511aaa89980310a578a6eccbc68a390b0767406a98c2ed863c2017978c: Status 404 returned error can't find the container with id 7e040a511aaa89980310a578a6eccbc68a390b0767406a98c2ed863c2017978c
Sep 16 18:47:23.779062 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:47:23.779033    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vmm77" event={"ID":"e388636a-eb1f-4148-b7df-dd7bff98deed","Type":"ContainerStarted","Data":"7e040a511aaa89980310a578a6eccbc68a390b0767406a98c2ed863c2017978c"}
Sep 16 18:47:25.787480 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:47:25.787441    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vmm77" event={"ID":"e388636a-eb1f-4148-b7df-dd7bff98deed","Type":"ContainerStarted","Data":"bc6af7c4a387fee140e32e2af374e6b5b4863703bd6f84b715516f3b2df81b5e"}
Sep 16 18:47:25.787907 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:47:25.787534    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vmm77" event={"ID":"e388636a-eb1f-4148-b7df-dd7bff98deed","Type":"ContainerStarted","Data":"26f144f212f0d4096bd34b3bea8121263ff97c12b1429807f04fc7f2312e2e58"}
Sep 16 18:47:25.787907 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:47:25.787603    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vmm77"
Sep 16 18:47:25.805947 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:47:25.805903    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vmm77" podStartSLOduration=137.204550777 podStartE2EDuration="2m18.80589061s" totalImagesPullingTime="1.601339833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:25.80444354 +0000 UTC m=+172.189002762" watchObservedRunningTime="2026-09-16 18:47:25.80589061 +0000 UTC m=+172.190449832"
Sep 16 18:47:35.793509 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:47:35.793480    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vmm77"
Sep 16 18:49:33.958841 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:49:33.958815    2570 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:50:56.587368 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:56.587332    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-94khl"]
Sep 16 18:50:56.589148 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:56.589133    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-94khl"
Sep 16 18:50:56.591404 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:56.591385    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-q6stj\""
Sep 16 18:50:56.592201 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:56.592183    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 18:50:56.592299 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:56.592236    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 18:50:56.605417 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:56.605399    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-94khl"]
Sep 16 18:50:56.685533 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:56.685503    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4zfw\" (UniqueName: \"kubernetes.io/projected/d958c271-a902-4e66-9907-e15d66b1e1e2-kube-api-access-n4zfw\") pod \"cert-manager-webhook-6486ffd997-94khl\" (UID: \"d958c271-a902-4e66-9907-e15d66b1e1e2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-94khl"
Sep 16 18:50:56.685731 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:56.685558    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d958c271-a902-4e66-9907-e15d66b1e1e2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-94khl\" (UID: \"d958c271-a902-4e66-9907-e15d66b1e1e2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-94khl"
Sep 16 18:50:56.786393 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:56.786353    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4zfw\" (UniqueName: \"kubernetes.io/projected/d958c271-a902-4e66-9907-e15d66b1e1e2-kube-api-access-n4zfw\") pod \"cert-manager-webhook-6486ffd997-94khl\" (UID: \"d958c271-a902-4e66-9907-e15d66b1e1e2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-94khl"
Sep 16 18:50:56.786539 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:56.786405    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d958c271-a902-4e66-9907-e15d66b1e1e2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-94khl\" (UID: \"d958c271-a902-4e66-9907-e15d66b1e1e2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-94khl"
Sep 16 18:50:56.819194 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:56.819161    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d958c271-a902-4e66-9907-e15d66b1e1e2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-94khl\" (UID: \"d958c271-a902-4e66-9907-e15d66b1e1e2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-94khl"
Sep 16 18:50:56.820218 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:56.820188    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4zfw\" (UniqueName: \"kubernetes.io/projected/d958c271-a902-4e66-9907-e15d66b1e1e2-kube-api-access-n4zfw\") pod \"cert-manager-webhook-6486ffd997-94khl\" (UID: \"d958c271-a902-4e66-9907-e15d66b1e1e2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-94khl"
Sep 16 18:50:56.897511 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:56.897442    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-94khl"
Sep 16 18:50:57.164636 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:57.164569    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-94khl"]
Sep 16 18:50:57.167097 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:50:57.167070    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd958c271_a902_4e66_9907_e15d66b1e1e2.slice/crio-86afb247986b5eb7cb2f73c3ac3532134d5a7388f0c56954511c8ea7ef082242 WatchSource:0}: Error finding container 86afb247986b5eb7cb2f73c3ac3532134d5a7388f0c56954511c8ea7ef082242: Status 404 returned error can't find the container with id 86afb247986b5eb7cb2f73c3ac3532134d5a7388f0c56954511c8ea7ef082242
Sep 16 18:50:57.169025 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:57.169008    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:50:57.435475 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:50:57.435417    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-94khl" event={"ID":"d958c271-a902-4e66-9907-e15d66b1e1e2","Type":"ContainerStarted","Data":"86afb247986b5eb7cb2f73c3ac3532134d5a7388f0c56954511c8ea7ef082242"}
Sep 16 18:51:00.448213 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:00.448140    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-94khl" event={"ID":"d958c271-a902-4e66-9907-e15d66b1e1e2","Type":"ContainerStarted","Data":"40bc97db1240a9b4baf93f2b07e275787522b4d0b0376fbe332581f3815e4001"}
Sep 16 18:51:00.448547 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:00.448256    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-94khl"
Sep 16 18:51:00.482175 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:00.482132    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-94khl" podStartSLOduration=1.517633162 podStartE2EDuration="4.482119584s" totalImagesPullingTime="2.964486422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:00.470754115 +0000 UTC m=+386.855313338" watchObservedRunningTime="2026-09-16 18:51:00.482119584 +0000 UTC m=+386.866678808"
Sep 16 18:51:05.896696 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:05.896653    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5d8xg"]
Sep 16 18:51:05.898484 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:05.898467    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5d8xg"
Sep 16 18:51:05.900999 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:05.900978    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-pxrds\""
Sep 16 18:51:05.908750 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:05.908730    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5d8xg"]
Sep 16 18:51:06.047212 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:06.047192    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/576067c3-ab21-4576-a182-6a6e549be937-bound-sa-token\") pod \"cert-manager-55b4d954d5-5d8xg\" (UID: \"576067c3-ab21-4576-a182-6a6e549be937\") " pod="cert-manager/cert-manager-55b4d954d5-5d8xg"
Sep 16 18:51:06.047314 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:06.047224    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6gv6\" (UniqueName: \"kubernetes.io/projected/576067c3-ab21-4576-a182-6a6e549be937-kube-api-access-g6gv6\") pod \"cert-manager-55b4d954d5-5d8xg\" (UID: \"576067c3-ab21-4576-a182-6a6e549be937\") " pod="cert-manager/cert-manager-55b4d954d5-5d8xg"
Sep 16 18:51:06.147898 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:06.147851    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/576067c3-ab21-4576-a182-6a6e549be937-bound-sa-token\") pod \"cert-manager-55b4d954d5-5d8xg\" (UID: \"576067c3-ab21-4576-a182-6a6e549be937\") " pod="cert-manager/cert-manager-55b4d954d5-5d8xg"
Sep 16 18:51:06.147898 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:06.147889    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6gv6\" (UniqueName: \"kubernetes.io/projected/576067c3-ab21-4576-a182-6a6e549be937-kube-api-access-g6gv6\") pod \"cert-manager-55b4d954d5-5d8xg\" (UID: \"576067c3-ab21-4576-a182-6a6e549be937\") " pod="cert-manager/cert-manager-55b4d954d5-5d8xg"
Sep 16 18:51:06.156805 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:06.156787    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/576067c3-ab21-4576-a182-6a6e549be937-bound-sa-token\") pod \"cert-manager-55b4d954d5-5d8xg\" (UID: \"576067c3-ab21-4576-a182-6a6e549be937\") " pod="cert-manager/cert-manager-55b4d954d5-5d8xg"
Sep 16 18:51:06.157088 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:06.157068    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6gv6\" (UniqueName: \"kubernetes.io/projected/576067c3-ab21-4576-a182-6a6e549be937-kube-api-access-g6gv6\") pod \"cert-manager-55b4d954d5-5d8xg\" (UID: \"576067c3-ab21-4576-a182-6a6e549be937\") " pod="cert-manager/cert-manager-55b4d954d5-5d8xg"
Sep 16 18:51:06.206880 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:06.206851    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5d8xg"
Sep 16 18:51:06.453377 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:06.453313    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-94khl"
Sep 16 18:51:06.475372 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:06.475351    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5d8xg"]
Sep 16 18:51:06.477752 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:51:06.477719    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod576067c3_ab21_4576_a182_6a6e549be937.slice/crio-995b1666e5e7fc6f44cfb93cc0bd0336198ecd58d93c26d5f90b3b569dc36277 WatchSource:0}: Error finding container 995b1666e5e7fc6f44cfb93cc0bd0336198ecd58d93c26d5f90b3b569dc36277: Status 404 returned error can't find the container with id 995b1666e5e7fc6f44cfb93cc0bd0336198ecd58d93c26d5f90b3b569dc36277
Sep 16 18:51:07.470214 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:07.470181    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5d8xg" event={"ID":"576067c3-ab21-4576-a182-6a6e549be937","Type":"ContainerStarted","Data":"c29e50225266c99914d8c2c8728ba39463345647fb3c04b6b61fcff28f0e0a92"}
Sep 16 18:51:07.470214 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:07.470220    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5d8xg" event={"ID":"576067c3-ab21-4576-a182-6a6e549be937","Type":"ContainerStarted","Data":"995b1666e5e7fc6f44cfb93cc0bd0336198ecd58d93c26d5f90b3b569dc36277"}
Sep 16 18:51:07.487916 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:07.487737    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-5d8xg" podStartSLOduration=2.487712307 podStartE2EDuration="2.487712307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:07.487590348 +0000 UTC m=+393.872149574" watchObservedRunningTime="2026-09-16 18:51:07.487712307 +0000 UTC m=+393.872271529"
Sep 16 18:51:36.123081 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.123046    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"]
Sep 16 18:51:36.127358 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.127336    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.132347 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.132326    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:51:36.132495 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.132477    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:51:36.132649 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.132358    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:51:36.133327 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.133306    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-q6tj2\""
Sep 16 18:51:36.145163 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.145143    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"]
Sep 16 18:51:36.245806 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.245782    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7508170-184a-47cc-9319-c53b95dccd82-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.245914 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.245809    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7508170-184a-47cc-9319-c53b95dccd82-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.245914 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.245828    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dpmh\" (UniqueName: \"kubernetes.io/projected/d7508170-184a-47cc-9319-c53b95dccd82-kube-api-access-9dpmh\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.245914 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.245849    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-istio-envoy\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.246087 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.245917    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-workload-socket\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.246087 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.245991    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-workload-certs\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.246087 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.246026    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-istio-data\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.246087 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.246053    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7508170-184a-47cc-9319-c53b95dccd82-istio-token\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.246230 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.246108    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-credential-socket\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.346518 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.346495    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7508170-184a-47cc-9319-c53b95dccd82-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.346621 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.346528    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7508170-184a-47cc-9319-c53b95dccd82-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.346621 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.346565    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dpmh\" (UniqueName: \"kubernetes.io/projected/d7508170-184a-47cc-9319-c53b95dccd82-kube-api-access-9dpmh\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.346770 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.346632    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-istio-envoy\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.346770 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.346660    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-workload-socket\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.346770 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.346703    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-workload-certs\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.346770 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.346728    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-istio-data\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.346770 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.346755    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7508170-184a-47cc-9319-c53b95dccd82-istio-token\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.347014 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.346783    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-credential-socket\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.347123 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.347100    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-istio-data\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.347173 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.347112    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-workload-certs\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.347173 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.347158    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-workload-socket\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.347247 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.347232    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7508170-184a-47cc-9319-c53b95dccd82-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.347366 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.347347    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-credential-socket\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.349034 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.349014    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7508170-184a-47cc-9319-c53b95dccd82-istio-envoy\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.349102 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.349087    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7508170-184a-47cc-9319-c53b95dccd82-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.356189 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.356160    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7508170-184a-47cc-9319-c53b95dccd82-istio-token\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.356585 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.356567    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dpmh\" (UniqueName: \"kubernetes.io/projected/d7508170-184a-47cc-9319-c53b95dccd82-kube-api-access-9dpmh\") pod \"openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j\" (UID: \"d7508170-184a-47cc-9319-c53b95dccd82\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.439913 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.439866    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:36.724788 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:36.724717    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"]
Sep 16 18:51:36.728182 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:51:36.728157    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7508170_184a_47cc_9319_c53b95dccd82.slice/crio-61b124e5daffd074b7a3aa44848155866b7b377f1faa7800e174738bf818545b WatchSource:0}: Error finding container 61b124e5daffd074b7a3aa44848155866b7b377f1faa7800e174738bf818545b: Status 404 returned error can't find the container with id 61b124e5daffd074b7a3aa44848155866b7b377f1faa7800e174738bf818545b
Sep 16 18:51:37.570117 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:37.570087    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j" event={"ID":"d7508170-184a-47cc-9319-c53b95dccd82","Type":"ContainerStarted","Data":"61b124e5daffd074b7a3aa44848155866b7b377f1faa7800e174738bf818545b"}
Sep 16 18:51:37.900719 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:37.900635    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"]
Sep 16 18:51:37.903577 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:37.903553    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:37.907422 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:37.906191    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 18:51:37.907422 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:37.906667    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mkdck\""
Sep 16 18:51:37.907422 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:37.906931    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 18:51:37.907422 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:37.907104    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 18:51:37.907422 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:37.907277    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 18:51:37.907751 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:37.907629    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:51:37.918843 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:37.918791    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"]
Sep 16 18:51:38.057485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:38.057450    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09d75669-3823-486c-961c-d9b0f3f499ac-manager-config\") pod \"lws-controller-manager-796cdc484c-ngwdv\" (UID: \"09d75669-3823-486c-961c-d9b0f3f499ac\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:38.057664 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:38.057500    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09d75669-3823-486c-961c-d9b0f3f499ac-cert\") pod \"lws-controller-manager-796cdc484c-ngwdv\" (UID: \"09d75669-3823-486c-961c-d9b0f3f499ac\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:38.057664 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:38.057529    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/09d75669-3823-486c-961c-d9b0f3f499ac-metrics-cert\") pod \"lws-controller-manager-796cdc484c-ngwdv\" (UID: \"09d75669-3823-486c-961c-d9b0f3f499ac\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:38.057800 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:38.057657    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vl9h\" (UniqueName: \"kubernetes.io/projected/09d75669-3823-486c-961c-d9b0f3f499ac-kube-api-access-4vl9h\") pod \"lws-controller-manager-796cdc484c-ngwdv\" (UID: \"09d75669-3823-486c-961c-d9b0f3f499ac\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:38.158622 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:38.158544    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vl9h\" (UniqueName: \"kubernetes.io/projected/09d75669-3823-486c-961c-d9b0f3f499ac-kube-api-access-4vl9h\") pod \"lws-controller-manager-796cdc484c-ngwdv\" (UID: \"09d75669-3823-486c-961c-d9b0f3f499ac\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:38.158622 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:38.158588    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09d75669-3823-486c-961c-d9b0f3f499ac-manager-config\") pod \"lws-controller-manager-796cdc484c-ngwdv\" (UID: \"09d75669-3823-486c-961c-d9b0f3f499ac\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:38.158854 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:38.158636    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09d75669-3823-486c-961c-d9b0f3f499ac-cert\") pod \"lws-controller-manager-796cdc484c-ngwdv\" (UID: \"09d75669-3823-486c-961c-d9b0f3f499ac\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:38.158854 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:38.158663    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/09d75669-3823-486c-961c-d9b0f3f499ac-metrics-cert\") pod \"lws-controller-manager-796cdc484c-ngwdv\" (UID: \"09d75669-3823-486c-961c-d9b0f3f499ac\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:38.159282 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:38.159256    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09d75669-3823-486c-961c-d9b0f3f499ac-manager-config\") pod \"lws-controller-manager-796cdc484c-ngwdv\" (UID: \"09d75669-3823-486c-961c-d9b0f3f499ac\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:38.161778 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:38.161751    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/09d75669-3823-486c-961c-d9b0f3f499ac-metrics-cert\") pod \"lws-controller-manager-796cdc484c-ngwdv\" (UID: \"09d75669-3823-486c-961c-d9b0f3f499ac\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:38.161894 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:38.161785    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09d75669-3823-486c-961c-d9b0f3f499ac-cert\") pod \"lws-controller-manager-796cdc484c-ngwdv\" (UID: \"09d75669-3823-486c-961c-d9b0f3f499ac\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:38.171047 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:38.171025    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vl9h\" (UniqueName: \"kubernetes.io/projected/09d75669-3823-486c-961c-d9b0f3f499ac-kube-api-access-4vl9h\") pod \"lws-controller-manager-796cdc484c-ngwdv\" (UID: \"09d75669-3823-486c-961c-d9b0f3f499ac\") " pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:38.216825 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:38.216797    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:38.500095 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:38.500022    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"]
Sep 16 18:51:39.460075 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:51:39.460003    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d75669_3823_486c_961c_d9b0f3f499ac.slice/crio-579c3973ad0230433b936d10bfff8f06db630310809841bfcb9d0b5b2a6c9232 WatchSource:0}: Error finding container 579c3973ad0230433b936d10bfff8f06db630310809841bfcb9d0b5b2a6c9232: Status 404 returned error can't find the container with id 579c3973ad0230433b936d10bfff8f06db630310809841bfcb9d0b5b2a6c9232
Sep 16 18:51:39.498245 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:39.498215    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:51:39.498313 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:39.498290    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:51:39.498346 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:39.498317    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:51:39.580245 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:39.580216    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv" event={"ID":"09d75669-3823-486c-961c-d9b0f3f499ac","Type":"ContainerStarted","Data":"579c3973ad0230433b936d10bfff8f06db630310809841bfcb9d0b5b2a6c9232"}
Sep 16 18:51:39.582007 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:39.581969    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j" event={"ID":"d7508170-184a-47cc-9319-c53b95dccd82","Type":"ContainerStarted","Data":"d9662259069a7f8312f56de76269fa82f527f593f6fb26957ae201dd34b00bd0"}
Sep 16 18:51:40.440004 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:40.439966    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:40.445158 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:40.445135    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:40.471951 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:40.471901    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j" podStartSLOduration=1.7040306219999999 podStartE2EDuration="4.471886341s" totalImagesPullingTime="2.767855719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:39.625184743 +0000 UTC m=+426.009743964" watchObservedRunningTime="2026-09-16 18:51:40.471886341 +0000 UTC m=+426.856445629"
Sep 16 18:51:40.585770 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:40.585699    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:40.587170 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:40.587147    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:51:41.590620 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:41.590588    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv" event={"ID":"09d75669-3823-486c-961c-d9b0f3f499ac","Type":"ContainerStarted","Data":"9e8a572aa305ee43921f4ec4d5b900270e64ad7d693782a41add6bc76da7eb6f"}
Sep 16 18:51:41.591020 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:41.590671    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:51:41.632497 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:41.632448    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv" podStartSLOduration=2.603536793 podStartE2EDuration="4.63243647s" totalImagesPullingTime="2.028899677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:41.629682774 +0000 UTC m=+428.014241998" watchObservedRunningTime="2026-09-16 18:51:41.63243647 +0000 UTC m=+428.016995692"
Sep 16 18:51:52.596788 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:51:52.596755    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-796cdc484c-ngwdv"
Sep 16 18:52:11.603919 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.603844    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"]
Sep 16 18:52:11.609810 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.609789    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"
Sep 16 18:52:11.612345 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.612326    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:52:11.612553 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.612538    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 18:52:11.612633 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.612539    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 18:52:11.613237 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.613218    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zzcxs\""
Sep 16 18:52:11.613316 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.613255    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:52:11.615512 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.615494    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"]
Sep 16 18:52:11.687959 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.687927    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znzwq\" (UniqueName: \"kubernetes.io/projected/617e873b-6589-470a-8a0f-f7d17e403bf4-kube-api-access-znzwq\") pod \"kuadrant-console-plugin-fbdd4f445-rf9nd\" (UID: \"617e873b-6589-470a-8a0f-f7d17e403bf4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"
Sep 16 18:52:11.688112 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.687967    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/617e873b-6589-470a-8a0f-f7d17e403bf4-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-rf9nd\" (UID: \"617e873b-6589-470a-8a0f-f7d17e403bf4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"
Sep 16 18:52:11.688112 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.688017    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/617e873b-6589-470a-8a0f-f7d17e403bf4-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rf9nd\" (UID: \"617e873b-6589-470a-8a0f-f7d17e403bf4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"
Sep 16 18:52:11.788755 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.788733    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/617e873b-6589-470a-8a0f-f7d17e403bf4-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-rf9nd\" (UID: \"617e873b-6589-470a-8a0f-f7d17e403bf4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"
Sep 16 18:52:11.788861 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.788776    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/617e873b-6589-470a-8a0f-f7d17e403bf4-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rf9nd\" (UID: \"617e873b-6589-470a-8a0f-f7d17e403bf4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"
Sep 16 18:52:11.788861 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.788813    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzwq\" (UniqueName: \"kubernetes.io/projected/617e873b-6589-470a-8a0f-f7d17e403bf4-kube-api-access-znzwq\") pod \"kuadrant-console-plugin-fbdd4f445-rf9nd\" (UID: \"617e873b-6589-470a-8a0f-f7d17e403bf4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"
Sep 16 18:52:11.788943 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:52:11.788931    2570 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 16 18:52:11.789014 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:52:11.789003    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/617e873b-6589-470a-8a0f-f7d17e403bf4-plugin-serving-cert podName:617e873b-6589-470a-8a0f-f7d17e403bf4 nodeName:}" failed. No retries permitted until 2026-09-16 18:52:12.288984788 +0000 UTC m=+458.673543990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/617e873b-6589-470a-8a0f-f7d17e403bf4-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-rf9nd" (UID: "617e873b-6589-470a-8a0f-f7d17e403bf4") : secret "plugin-serving-cert" not found
Sep 16 18:52:11.789318 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.789301    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/617e873b-6589-470a-8a0f-f7d17e403bf4-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-rf9nd\" (UID: \"617e873b-6589-470a-8a0f-f7d17e403bf4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"
Sep 16 18:52:11.802290 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:11.802271    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzwq\" (UniqueName: \"kubernetes.io/projected/617e873b-6589-470a-8a0f-f7d17e403bf4-kube-api-access-znzwq\") pod \"kuadrant-console-plugin-fbdd4f445-rf9nd\" (UID: \"617e873b-6589-470a-8a0f-f7d17e403bf4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"
Sep 16 18:52:12.292389 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:12.292360    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/617e873b-6589-470a-8a0f-f7d17e403bf4-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rf9nd\" (UID: \"617e873b-6589-470a-8a0f-f7d17e403bf4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"
Sep 16 18:52:12.294632 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:12.294606    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/617e873b-6589-470a-8a0f-f7d17e403bf4-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-rf9nd\" (UID: \"617e873b-6589-470a-8a0f-f7d17e403bf4\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"
Sep 16 18:52:12.519261 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:12.519236    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"
Sep 16 18:52:12.786558 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:12.786530    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd"]
Sep 16 18:52:12.789326 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:52:12.789290    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod617e873b_6589_470a_8a0f_f7d17e403bf4.slice/crio-6cf4cb50955907be2f736823df89b595889bc8bdec0df37b03c6bf77c8eb45ac WatchSource:0}: Error finding container 6cf4cb50955907be2f736823df89b595889bc8bdec0df37b03c6bf77c8eb45ac: Status 404 returned error can't find the container with id 6cf4cb50955907be2f736823df89b595889bc8bdec0df37b03c6bf77c8eb45ac
Sep 16 18:52:13.710800 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:13.707061    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd" event={"ID":"617e873b-6589-470a-8a0f-f7d17e403bf4","Type":"ContainerStarted","Data":"6cf4cb50955907be2f736823df89b595889bc8bdec0df37b03c6bf77c8eb45ac"}
Sep 16 18:52:17.723943 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:17.723905    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd" event={"ID":"617e873b-6589-470a-8a0f-f7d17e403bf4","Type":"ContainerStarted","Data":"d3050adb3d0ca980cf588cf1dce8fea1b26bc8cc65541171185854a7db219676"}
Sep 16 18:52:17.739729 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:17.739658    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-rf9nd" podStartSLOduration=2.46082128 podStartE2EDuration="6.739642384s" totalImagesPullingTime="4.278821104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:17.738820586 +0000 UTC m=+464.123379810" watchObservedRunningTime="2026-09-16 18:52:17.739642384 +0000 UTC m=+464.124201607"
Sep 16 18:52:56.433483 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:56.433446    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v89xw"]
Sep 16 18:52:56.451027 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:56.450999    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v89xw"]
Sep 16 18:52:56.451182 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:56.451085    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-v89xw"
Sep 16 18:52:56.453478 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:56.453449    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 18:52:56.461884 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:56.461863    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v89xw"]
Sep 16 18:52:56.597631 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:56.597597    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdbc426f-12b9-481e-b689-58a0a7b8dc3c-config-file\") pod \"limitador-limitador-84994f7ffc-v89xw\" (UID: \"bdbc426f-12b9-481e-b689-58a0a7b8dc3c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v89xw"
Sep 16 18:52:56.597785 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:56.597643    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdvds\" (UniqueName: \"kubernetes.io/projected/bdbc426f-12b9-481e-b689-58a0a7b8dc3c-kube-api-access-vdvds\") pod \"limitador-limitador-84994f7ffc-v89xw\" (UID: \"bdbc426f-12b9-481e-b689-58a0a7b8dc3c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v89xw"
Sep 16 18:52:56.698648 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:56.698583    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdbc426f-12b9-481e-b689-58a0a7b8dc3c-config-file\") pod \"limitador-limitador-84994f7ffc-v89xw\" (UID: \"bdbc426f-12b9-481e-b689-58a0a7b8dc3c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v89xw"
Sep 16 18:52:56.698648 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:56.698634    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdvds\" (UniqueName: \"kubernetes.io/projected/bdbc426f-12b9-481e-b689-58a0a7b8dc3c-kube-api-access-vdvds\") pod \"limitador-limitador-84994f7ffc-v89xw\" (UID: \"bdbc426f-12b9-481e-b689-58a0a7b8dc3c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v89xw"
Sep 16 18:52:56.699189 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:56.699168    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdbc426f-12b9-481e-b689-58a0a7b8dc3c-config-file\") pod \"limitador-limitador-84994f7ffc-v89xw\" (UID: \"bdbc426f-12b9-481e-b689-58a0a7b8dc3c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v89xw"
Sep 16 18:52:56.708023 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:56.708005    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdvds\" (UniqueName: \"kubernetes.io/projected/bdbc426f-12b9-481e-b689-58a0a7b8dc3c-kube-api-access-vdvds\") pod \"limitador-limitador-84994f7ffc-v89xw\" (UID: \"bdbc426f-12b9-481e-b689-58a0a7b8dc3c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v89xw"
Sep 16 18:52:56.760940 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:56.760913    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-v89xw"
Sep 16 18:52:57.027400 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:57.027373    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v89xw"]
Sep 16 18:52:57.029971 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:52:57.029947    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbc426f_12b9_481e_b689_58a0a7b8dc3c.slice/crio-80ac0688e96c7d94db70b03e59005f6bd7428f5e77b882151d1907441cd1247f WatchSource:0}: Error finding container 80ac0688e96c7d94db70b03e59005f6bd7428f5e77b882151d1907441cd1247f: Status 404 returned error can't find the container with id 80ac0688e96c7d94db70b03e59005f6bd7428f5e77b882151d1907441cd1247f
Sep 16 18:52:57.857608 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:52:57.857573    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-v89xw" event={"ID":"bdbc426f-12b9-481e-b689-58a0a7b8dc3c","Type":"ContainerStarted","Data":"80ac0688e96c7d94db70b03e59005f6bd7428f5e77b882151d1907441cd1247f"}
Sep 16 18:53:01.874804 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:53:01.874771    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-v89xw" event={"ID":"bdbc426f-12b9-481e-b689-58a0a7b8dc3c","Type":"ContainerStarted","Data":"62f6c633df6a5d541f1d8e00acee639bd156e47f27c62186c41d958183363761"}
Sep 16 18:53:01.875172 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:53:01.874889    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-v89xw"
Sep 16 18:53:01.894665 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:53:01.894611    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-v89xw" podStartSLOduration=1.681560479 podStartE2EDuration="5.894597073s" totalImagesPullingTime="4.213036594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:01.893784198 +0000 UTC m=+508.278343423" watchObservedRunningTime="2026-09-16 18:53:01.894597073 +0000 UTC m=+508.279156295"
Sep 16 18:53:12.879494 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:53:12.879464    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-v89xw"
Sep 16 18:57:21.748346 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:21.748315    2570 generic.go:356] "Generic (PLEG): container finished" podID="d7508170-184a-47cc-9319-c53b95dccd82" containerID="d9662259069a7f8312f56de76269fa82f527f593f6fb26957ae201dd34b00bd0" exitCode=0
Sep 16 18:57:21.748742 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:21.748376    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j" event={"ID":"d7508170-184a-47cc-9319-c53b95dccd82","Type":"ContainerDied","Data":"d9662259069a7f8312f56de76269fa82f527f593f6fb26957ae201dd34b00bd0"}
Sep 16 18:57:21.748742 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:21.748736    2570 scope.go:117] "RemoveContainer" containerID="d9662259069a7f8312f56de76269fa82f527f593f6fb26957ae201dd34b00bd0"
Sep 16 18:57:21.749559 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:21.749533    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:57:21.749617 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:21.749597    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:57:21.749653 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:21.749629    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:57:22.756982 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:22.756945    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j" event={"ID":"d7508170-184a-47cc-9319-c53b95dccd82","Type":"ContainerStarted","Data":"b3db6ca147cd67d91f1e0b00794d17ec0ef46c71ab3587ece7fe6d96cd6e991f"}
Sep 16 18:57:23.441078 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:23.441048    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:57:23.442076 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:23.442044    2570 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.15:15021/healthz/ready\": dial tcp 10.132.0.15:15021: connect: connection refused" start-of-body=
Sep 16 18:57:23.442269 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:23.442243    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j" podUID="d7508170-184a-47cc-9319-c53b95dccd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.15:15021/healthz/ready\": dial tcp 10.132.0.15:15021: connect: connection refused"
Sep 16 18:57:24.440569 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.440521    2570 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.15:15021/healthz/ready\": dial tcp 10.132.0.15:15021: connect: connection refused" start-of-body=
Sep 16 18:57:24.441079 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.440591    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j" podUID="d7508170-184a-47cc-9319-c53b95dccd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.15:15021/healthz/ready\": dial tcp 10.132.0.15:15021: connect: connection refused"
Sep 16 18:57:24.695923 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.695843    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:57:24.697909 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.697892    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.700720 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.700679    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-nmghz\""
Sep 16 18:57:24.701050 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.701024    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:57:24.701521 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.701496    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-b74jj\""
Sep 16 18:57:24.701720 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.701680    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 18:57:24.701841 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.701708    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 18:57:24.701934 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.701771    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 18:57:24.705806 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.705786    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 18:57:24.715829 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.715806    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:57:24.794126 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.794071    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.794270 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.794132    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.794270 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.794177    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.794342 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.794265    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657b1ca3-7038-4b50-aa10-15f640a4b4ef-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.794342 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.794326    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjcz5\" (UniqueName: \"kubernetes.io/projected/657b1ca3-7038-4b50-aa10-15f640a4b4ef-kube-api-access-zjcz5\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.794411 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.794353    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.794411 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.794393    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.794480 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.794431    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/657b1ca3-7038-4b50-aa10-15f640a4b4ef-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.895652 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.895614    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.895652 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.895656    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.895861 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.895678    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.895861 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.895724    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657b1ca3-7038-4b50-aa10-15f640a4b4ef-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.895861 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.895775    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjcz5\" (UniqueName: \"kubernetes.io/projected/657b1ca3-7038-4b50-aa10-15f640a4b4ef-kube-api-access-zjcz5\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.895861 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.895804    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.895861 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.895839    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.896082 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.895863    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/657b1ca3-7038-4b50-aa10-15f640a4b4ef-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.896082 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.896051    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.896209 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.896185    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.896270 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.896253    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.896348 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.896327    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.896606 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.896588    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/657b1ca3-7038-4b50-aa10-15f640a4b4ef-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.898162 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.898145    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.898201 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.898154    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657b1ca3-7038-4b50-aa10-15f640a4b4ef-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:24.904755 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:24.904734    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjcz5\" (UniqueName: \"kubernetes.io/projected/657b1ca3-7038-4b50-aa10-15f640a4b4ef-kube-api-access-zjcz5\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:25.008006 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:25.007929    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:57:25.305521 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:25.305491    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:57:25.308630 ip-10-0-135-244 kubenswrapper[2570]: W0916 18:57:25.308595    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657b1ca3_7038_4b50_aa10_15f640a4b4ef.slice/crio-5a69de89f4be621bdf09e7d11b39dfa319199ebbdc7a06060aeaf49fcbd00410 WatchSource:0}: Error finding container 5a69de89f4be621bdf09e7d11b39dfa319199ebbdc7a06060aeaf49fcbd00410: Status 404 returned error can't find the container with id 5a69de89f4be621bdf09e7d11b39dfa319199ebbdc7a06060aeaf49fcbd00410
Sep 16 18:57:25.310890 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:25.310872    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:57:25.440951 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:25.440917    2570 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.15:15021/healthz/ready\": dial tcp 10.132.0.15:15021: connect: connection refused" start-of-body=
Sep 16 18:57:25.441370 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:25.440980    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j" podUID="d7508170-184a-47cc-9319-c53b95dccd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.15:15021/healthz/ready\": dial tcp 10.132.0.15:15021: connect: connection refused"
Sep 16 18:57:25.586370 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:25.586292    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:57:25.779178 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:25.779111    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"657b1ca3-7038-4b50-aa10-15f640a4b4ef","Type":"ContainerStarted","Data":"5a69de89f4be621bdf09e7d11b39dfa319199ebbdc7a06060aeaf49fcbd00410"}
Sep 16 18:57:26.459718 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:26.459372    2570 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.15:15021/healthz/ready\": dial tcp 10.132.0.15:15021: connect: connection refused" start-of-body=
Sep 16 18:57:26.459718 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:26.459434    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j" podUID="d7508170-184a-47cc-9319-c53b95dccd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.15:15021/healthz/ready\": dial tcp 10.132.0.15:15021: connect: connection refused"
Sep 16 18:57:27.456498 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:27.456035    2570 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.15:15021/healthz/ready\": dial tcp 10.132.0.15:15021: connect: connection refused" start-of-body=
Sep 16 18:57:27.456498 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:27.456104    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j" podUID="d7508170-184a-47cc-9319-c53b95dccd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.15:15021/healthz/ready\": dial tcp 10.132.0.15:15021: connect: connection refused"
Sep 16 18:57:28.445264 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:28.445230    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:57:28.446236 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:28.446214    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j"
Sep 16 18:57:29.798020 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:29.797980    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"657b1ca3-7038-4b50-aa10-15f640a4b4ef","Type":"ContainerStarted","Data":"0884dec18391363bb17f8e341683499f151d89513816f360d2b573d15c32fbce"}
Sep 16 18:57:31.807445 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:31.807413    2570 generic.go:356] "Generic (PLEG): container finished" podID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerID="0884dec18391363bb17f8e341683499f151d89513816f360d2b573d15c32fbce" exitCode=0
Sep 16 18:57:31.807841 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:31.807453    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"657b1ca3-7038-4b50-aa10-15f640a4b4ef","Type":"ContainerDied","Data":"0884dec18391363bb17f8e341683499f151d89513816f360d2b573d15c32fbce"}
Sep 16 18:57:59.931898 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:59.931859    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"657b1ca3-7038-4b50-aa10-15f640a4b4ef","Type":"ContainerStarted","Data":"4441f746517903e5b88be7051cb9e05ec290ec18c28d82e26bad2ca2c4a6cd6f"}
Sep 16 18:57:59.955118 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:57:59.955056    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.924999983 podStartE2EDuration="35.955034422s" totalImagesPullingTime="30.951972481s" totalInitContainerRuntime="2.078061958s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:59.953514753 +0000 UTC m=+806.338073977" watchObservedRunningTime="2026-09-16 18:57:59.955034422 +0000 UTC m=+806.339593646"
Sep 16 18:58:00.008464 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:58:00.008434    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:05.008415 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:58:05.008383    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:05.010067 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:58:05.010031    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.19:8000/health\": dial tcp 10.132.0.19:8000: connect: connection refused"
Sep 16 18:58:15.008827 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:58:15.008780    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.19:8000/health\": dial tcp 10.132.0.19:8000: connect: connection refused"
Sep 16 18:58:25.008602 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:58:25.008560    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.19:8000/health\": dial tcp 10.132.0.19:8000: connect: connection refused"
Sep 16 18:58:35.009317 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:58:35.009276    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.19:8000/health\": dial tcp 10.132.0.19:8000: connect: connection refused"
Sep 16 18:58:45.009020 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:58:45.008969    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.19:8000/health\": dial tcp 10.132.0.19:8000: connect: connection refused"
Sep 16 18:58:55.009009 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:58:55.008965    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.19:8000/health\": dial tcp 10.132.0.19:8000: connect: connection refused"
Sep 16 18:59:05.008532 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:05.008485    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.19:8000/health\": dial tcp 10.132.0.19:8000: connect: connection refused"
Sep 16 18:59:15.008954 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:15.008911    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.19:8000/health\": dial tcp 10.132.0.19:8000: connect: connection refused"
Sep 16 18:59:25.008596 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:25.008543    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="main" probeResult="failure" output="Get \"https://10.132.0.19:8000/health\": dial tcp 10.132.0.19:8000: connect: connection refused"
Sep 16 18:59:35.017988 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:35.017960    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:59:35.025385 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:35.025366    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:59:40.262023 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:40.261989    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:59:55.283491 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.283430    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="main" containerID="cri-o://4441f746517903e5b88be7051cb9e05ec290ec18c28d82e26bad2ca2c4a6cd6f" gracePeriod=45
Sep 16 18:59:55.937222 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.937193    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:59:55.978829 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.978795    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/657b1ca3-7038-4b50-aa10-15f640a4b4ef-cabundle-cert\") pod \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") "
Sep 16 18:59:55.978981 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.978840    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-dshm\") pod \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") "
Sep 16 18:59:55.978981 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.978866    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-tmp-dir\") pod \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") "
Sep 16 18:59:55.978981 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.978888    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-kserve-provision-location\") pod \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") "
Sep 16 18:59:55.979142 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.979035    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjcz5\" (UniqueName: \"kubernetes.io/projected/657b1ca3-7038-4b50-aa10-15f640a4b4ef-kube-api-access-zjcz5\") pod \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") "
Sep 16 18:59:55.979142 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.979071    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-home\") pod \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") "
Sep 16 18:59:55.979240 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.979153    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657b1ca3-7038-4b50-aa10-15f640a4b4ef-tls-certs\") pod \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") "
Sep 16 18:59:55.979240 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.979196    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-model-cache\") pod \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\" (UID: \"657b1ca3-7038-4b50-aa10-15f640a4b4ef\") "
Sep 16 18:59:55.979638 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.979599    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "657b1ca3-7038-4b50-aa10-15f640a4b4ef" (UID: "657b1ca3-7038-4b50-aa10-15f640a4b4ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:55.979792 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.979719    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657b1ca3-7038-4b50-aa10-15f640a4b4ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "657b1ca3-7038-4b50-aa10-15f640a4b4ef" (UID: "657b1ca3-7038-4b50-aa10-15f640a4b4ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:55.980159 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.980118    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-home" (OuterVolumeSpecName: "home") pod "657b1ca3-7038-4b50-aa10-15f640a4b4ef" (UID: "657b1ca3-7038-4b50-aa10-15f640a4b4ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:55.981730 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.981700    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/657b1ca3-7038-4b50-aa10-15f640a4b4ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "657b1ca3-7038-4b50-aa10-15f640a4b4ef" (UID: "657b1ca3-7038-4b50-aa10-15f640a4b4ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:55.981730 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.981721    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-dshm" (OuterVolumeSpecName: "dshm") pod "657b1ca3-7038-4b50-aa10-15f640a4b4ef" (UID: "657b1ca3-7038-4b50-aa10-15f640a4b4ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:55.981887 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.981869    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657b1ca3-7038-4b50-aa10-15f640a4b4ef-kube-api-access-zjcz5" (OuterVolumeSpecName: "kube-api-access-zjcz5") pod "657b1ca3-7038-4b50-aa10-15f640a4b4ef" (UID: "657b1ca3-7038-4b50-aa10-15f640a4b4ef"). InnerVolumeSpecName "kube-api-access-zjcz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:55.991715 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:55.991677    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "657b1ca3-7038-4b50-aa10-15f640a4b4ef" (UID: "657b1ca3-7038-4b50-aa10-15f640a4b4ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:56.004485 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.004458    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "657b1ca3-7038-4b50-aa10-15f640a4b4ef" (UID: "657b1ca3-7038-4b50-aa10-15f640a4b4ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:56.080486 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.080436    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/657b1ca3-7038-4b50-aa10-15f640a4b4ef-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.080486 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.080457    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.080486 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.080465    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/657b1ca3-7038-4b50-aa10-15f640a4b4ef-cabundle-cert\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.080486 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.080474    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.080486 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.080482    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.080663 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.080490    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-kserve-provision-location\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.080663 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.080499    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjcz5\" (UniqueName: \"kubernetes.io/projected/657b1ca3-7038-4b50-aa10-15f640a4b4ef-kube-api-access-zjcz5\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.080663 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.080508    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/657b1ca3-7038-4b50-aa10-15f640a4b4ef-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.361833 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.361760    2570 generic.go:356] "Generic (PLEG): container finished" podID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerID="4441f746517903e5b88be7051cb9e05ec290ec18c28d82e26bad2ca2c4a6cd6f" exitCode=0
Sep 16 18:59:56.361833 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.361799    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"657b1ca3-7038-4b50-aa10-15f640a4b4ef","Type":"ContainerDied","Data":"4441f746517903e5b88be7051cb9e05ec290ec18c28d82e26bad2ca2c4a6cd6f"}
Sep 16 18:59:56.361833 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.361803    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:59:56.362295 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.361843    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"657b1ca3-7038-4b50-aa10-15f640a4b4ef","Type":"ContainerDied","Data":"5a69de89f4be621bdf09e7d11b39dfa319199ebbdc7a06060aeaf49fcbd00410"}
Sep 16 18:59:56.362295 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.361871    2570 scope.go:117] "RemoveContainer" containerID="4441f746517903e5b88be7051cb9e05ec290ec18c28d82e26bad2ca2c4a6cd6f"
Sep 16 18:59:56.369955 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.369934    2570 scope.go:117] "RemoveContainer" containerID="0884dec18391363bb17f8e341683499f151d89513816f360d2b573d15c32fbce"
Sep 16 18:59:56.377274 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.377256    2570 scope.go:117] "RemoveContainer" containerID="4441f746517903e5b88be7051cb9e05ec290ec18c28d82e26bad2ca2c4a6cd6f"
Sep 16 18:59:56.377521 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:59:56.377499    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4441f746517903e5b88be7051cb9e05ec290ec18c28d82e26bad2ca2c4a6cd6f\": container with ID starting with 4441f746517903e5b88be7051cb9e05ec290ec18c28d82e26bad2ca2c4a6cd6f not found: ID does not exist" containerID="4441f746517903e5b88be7051cb9e05ec290ec18c28d82e26bad2ca2c4a6cd6f"
Sep 16 18:59:56.377597 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.377532    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4441f746517903e5b88be7051cb9e05ec290ec18c28d82e26bad2ca2c4a6cd6f"} err="failed to get container status \"4441f746517903e5b88be7051cb9e05ec290ec18c28d82e26bad2ca2c4a6cd6f\": rpc error: code = NotFound desc = could not find container \"4441f746517903e5b88be7051cb9e05ec290ec18c28d82e26bad2ca2c4a6cd6f\": container with ID starting with 4441f746517903e5b88be7051cb9e05ec290ec18c28d82e26bad2ca2c4a6cd6f not found: ID does not exist"
Sep 16 18:59:56.377597 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.377555    2570 scope.go:117] "RemoveContainer" containerID="0884dec18391363bb17f8e341683499f151d89513816f360d2b573d15c32fbce"
Sep 16 18:59:56.377908 ip-10-0-135-244 kubenswrapper[2570]: E0916 18:59:56.377876    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0884dec18391363bb17f8e341683499f151d89513816f360d2b573d15c32fbce\": container with ID starting with 0884dec18391363bb17f8e341683499f151d89513816f360d2b573d15c32fbce not found: ID does not exist" containerID="0884dec18391363bb17f8e341683499f151d89513816f360d2b573d15c32fbce"
Sep 16 18:59:56.378010 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.377915    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0884dec18391363bb17f8e341683499f151d89513816f360d2b573d15c32fbce"} err="failed to get container status \"0884dec18391363bb17f8e341683499f151d89513816f360d2b573d15c32fbce\": rpc error: code = NotFound desc = could not find container \"0884dec18391363bb17f8e341683499f151d89513816f360d2b573d15c32fbce\": container with ID starting with 0884dec18391363bb17f8e341683499f151d89513816f360d2b573d15c32fbce not found: ID does not exist"
Sep 16 18:59:56.379661 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.379639    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:59:56.384989 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:56.384970    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:59:58.160133 ip-10-0-135-244 kubenswrapper[2570]: I0916 18:59:58.160099    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" path="/var/lib/kubelet/pods/657b1ca3-7038-4b50-aa10-15f640a4b4ef/volumes"
Sep 16 19:00:36.950506 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:36.950474    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"]
Sep 16 19:00:36.951001 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:36.950864    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="main"
Sep 16 19:00:36.951001 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:36.950882    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="main"
Sep 16 19:00:36.951001 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:36.950901    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="storage-initializer"
Sep 16 19:00:36.951001 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:36.950910    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="storage-initializer"
Sep 16 19:00:36.951001 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:36.950967    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="657b1ca3-7038-4b50-aa10-15f640a4b4ef" containerName="main"
Sep 16 19:00:36.954864 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:36.954843    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:36.958392 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:36.958368    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 19:00:36.959394 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:36.959375    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 19:00:36.959488 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:36.959406    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 19:00:36.959488 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:36.959415    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-mlvl2\""
Sep 16 19:00:36.971837 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:36.971816    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"]
Sep 16 19:00:37.057832 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.057801    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.057939 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.057843    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.057939 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.057927    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.058028 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.057955    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.058028 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.057988    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.058105 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.058037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsq55\" (UniqueName: \"kubernetes.io/projected/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-kube-api-access-bsq55\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.158635 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.158605    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.158743 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.158640    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.158743 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.158682    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.158743 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.158723    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.158913 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.158761    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.158913 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.158796    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsq55\" (UniqueName: \"kubernetes.io/projected/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-kube-api-access-bsq55\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.159090 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.159068    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.159150 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.159112    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.159200 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.159144    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.160834 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.160815    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.161110 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.161092    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.167791 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.167766    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsq55\" (UniqueName: \"kubernetes.io/projected/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-kube-api-access-bsq55\") pod \"scheduler-inline-config-test-kserve-8655bc757-6mdm5\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.265319 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.265262    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:37.538908 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:37.538883    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"]
Sep 16 19:00:37.540741 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:00:37.540713    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d5a451_8cc5_4810_a80e_8b1b5567e7e8.slice/crio-0130a44d3eeb9c5b46b13a36d118eef747921eba4a32fe7e70373f0be49927db WatchSource:0}: Error finding container 0130a44d3eeb9c5b46b13a36d118eef747921eba4a32fe7e70373f0be49927db: Status 404 returned error can't find the container with id 0130a44d3eeb9c5b46b13a36d118eef747921eba4a32fe7e70373f0be49927db
Sep 16 19:00:38.510314 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:38.510271    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5" event={"ID":"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8","Type":"ContainerStarted","Data":"0130a44d3eeb9c5b46b13a36d118eef747921eba4a32fe7e70373f0be49927db"}
Sep 16 19:00:39.515442 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:39.515402    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5" event={"ID":"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8","Type":"ContainerStarted","Data":"149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08"}
Sep 16 19:00:40.265481 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:40.265450    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:47.266088 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:47.266045    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:47.278426 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:47.278401    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:00:47.298103 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:47.298030    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5" podStartSLOduration=9.916644448 podStartE2EDuration="11.298016583s" totalImagesPullingTime="1.381372135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:39.560529606 +0000 UTC m=+965.945088829" watchObservedRunningTime="2026-09-16 19:00:47.298016583 +0000 UTC m=+973.682575805"
Sep 16 19:00:47.554553 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:00:47.554530    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:01:28.564388 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:28.564358    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"]
Sep 16 19:01:28.574471 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:01:28.574435    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:01:28.574471 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:28.574471 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:28.574471 ip-10-0-135-244 kubenswrapper[2570]:  > containerID="149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08" cmd=["/bin/sleep","15"]
Sep 16 19:01:28.574771 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:28.574489    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5" message="" err=<
Sep 16 19:01:28.574771 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:28.574771 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:28.574771 ip-10-0-135-244 kubenswrapper[2570]:  >
Sep 16 19:01:28.574771 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:01:28.574501    2570 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:01:28.574771 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:28.574771 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:28.574771 ip-10-0-135-244 kubenswrapper[2570]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5" podUID="d4d5a451-8cc5-4810-a80e-8b1b5567e7e8" containerName="main" containerID="cri-o://149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08"
Sep 16 19:01:28.574771 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:28.574543    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5" podUID="d4d5a451-8cc5-4810-a80e-8b1b5567e7e8" containerName="main" containerID="cri-o://149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08" gracePeriod=60
Sep 16 19:01:28.907468 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:28.907446    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:01:29.038545 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.038514    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-tls-certs\") pod \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") "
Sep 16 19:01:29.038675 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.038553    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-dshm\") pod \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") "
Sep 16 19:01:29.038675 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.038590    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-tmp-dir\") pod \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") "
Sep 16 19:01:29.038675 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.038613    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-home\") pod \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") "
Sep 16 19:01:29.038675 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.038641    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsq55\" (UniqueName: \"kubernetes.io/projected/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-kube-api-access-bsq55\") pod \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") "
Sep 16 19:01:29.038675 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.038666    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-model-cache\") pod \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\" (UID: \"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8\") "
Sep 16 19:01:29.038922 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.038900    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-home" (OuterVolumeSpecName: "home") pod "d4d5a451-8cc5-4810-a80e-8b1b5567e7e8" (UID: "d4d5a451-8cc5-4810-a80e-8b1b5567e7e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:29.038980 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.038955    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4d5a451-8cc5-4810-a80e-8b1b5567e7e8" (UID: "d4d5a451-8cc5-4810-a80e-8b1b5567e7e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:29.039086 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.039042    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4d5a451-8cc5-4810-a80e-8b1b5567e7e8" (UID: "d4d5a451-8cc5-4810-a80e-8b1b5567e7e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:29.040895 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.040870    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4d5a451-8cc5-4810-a80e-8b1b5567e7e8" (UID: "d4d5a451-8cc5-4810-a80e-8b1b5567e7e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:29.040997 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.040889    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-kube-api-access-bsq55" (OuterVolumeSpecName: "kube-api-access-bsq55") pod "d4d5a451-8cc5-4810-a80e-8b1b5567e7e8" (UID: "d4d5a451-8cc5-4810-a80e-8b1b5567e7e8"). InnerVolumeSpecName "kube-api-access-bsq55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:29.040997 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.040952    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-dshm" (OuterVolumeSpecName: "dshm") pod "d4d5a451-8cc5-4810-a80e-8b1b5567e7e8" (UID: "d4d5a451-8cc5-4810-a80e-8b1b5567e7e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:29.139410 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.139344    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:01:29.139410 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.139367    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:01:29.139410 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.139401    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:01:29.139568 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.139413    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:01:29.139568 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.139426    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsq55\" (UniqueName: \"kubernetes.io/projected/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-kube-api-access-bsq55\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:01:29.139568 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.139438    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:01:29.692967 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.692884    2570 generic.go:356] "Generic (PLEG): container finished" podID="d4d5a451-8cc5-4810-a80e-8b1b5567e7e8" containerID="149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08" exitCode=0
Sep 16 19:01:29.692967 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.692928    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"
Sep 16 19:01:29.693382 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.692973    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5" event={"ID":"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8","Type":"ContainerDied","Data":"149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08"}
Sep 16 19:01:29.693382 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.693027    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5" event={"ID":"d4d5a451-8cc5-4810-a80e-8b1b5567e7e8","Type":"ContainerDied","Data":"0130a44d3eeb9c5b46b13a36d118eef747921eba4a32fe7e70373f0be49927db"}
Sep 16 19:01:29.693382 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.693051    2570 scope.go:117] "RemoveContainer" containerID="149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08"
Sep 16 19:01:29.701192 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.701172    2570 scope.go:117] "RemoveContainer" containerID="149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08"
Sep 16 19:01:29.701490 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:01:29.701468    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08\": container with ID starting with 149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08 not found: ID does not exist" containerID="149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08"
Sep 16 19:01:29.701540 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.701498    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08"} err="failed to get container status \"149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08\": rpc error: code = NotFound desc = could not find container \"149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08\": container with ID starting with 149727d633461afe0887edf8e91f7638768f5051c440ce8585ecf511c44e6a08 not found: ID does not exist"
Sep 16 19:01:29.713529 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.713508    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"]
Sep 16 19:01:29.721968 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:29.719435    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6mdm5"]
Sep 16 19:01:30.160641 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:01:30.160606    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4d5a451-8cc5-4810-a80e-8b1b5567e7e8" path="/var/lib/kubelet/pods/d4d5a451-8cc5-4810-a80e-8b1b5567e7e8/volumes"
Sep 16 19:02:26.055457 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.055420    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"]
Sep 16 19:02:26.055847 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.055646    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4d5a451-8cc5-4810-a80e-8b1b5567e7e8" containerName="main"
Sep 16 19:02:26.055847 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.055656    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4d5a451-8cc5-4810-a80e-8b1b5567e7e8" containerName="main"
Sep 16 19:02:26.055847 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.055730    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4d5a451-8cc5-4810-a80e-8b1b5567e7e8" containerName="main"
Sep 16 19:02:26.058404 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.058386    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.060828 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.060804    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-dmjt9\""
Sep 16 19:02:26.061598 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.061577    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 19:02:26.061598 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.061589    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 19:02:26.062000 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.061985    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 19:02:26.074144 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.074125    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"]
Sep 16 19:02:26.231064 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.231037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.231198 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.231082    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.231198 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.231139    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.231198 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.231174    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.231301 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.231209    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3c4403-74d8-4de4-addc-c19380e16611-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.231301 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.231230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2s4k\" (UniqueName: \"kubernetes.io/projected/7c3c4403-74d8-4de4-addc-c19380e16611-kube-api-access-c2s4k\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.331552 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.331485    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3c4403-74d8-4de4-addc-c19380e16611-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.331552 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.331518    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2s4k\" (UniqueName: \"kubernetes.io/projected/7c3c4403-74d8-4de4-addc-c19380e16611-kube-api-access-c2s4k\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.331552 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.331538    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.331834 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.331573    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.331834 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.331739    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.331834 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.331786    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.331985 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.331893    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.332038 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.332021    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.332135 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.332116    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.333779 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.333753    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.334006 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.333989    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3c4403-74d8-4de4-addc-c19380e16611-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.343105 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.343085    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2s4k\" (UniqueName: \"kubernetes.io/projected/7c3c4403-74d8-4de4-addc-c19380e16611-kube-api-access-c2s4k\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.368070 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.368043    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:26.641798 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.641729    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"]
Sep 16 19:02:26.645099 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:02:26.645072    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3c4403_74d8_4de4_addc_c19380e16611.slice/crio-1637ded6e19336c38b5978b0d1ce591355776953f7d022206ca6569bfe841bbd WatchSource:0}: Error finding container 1637ded6e19336c38b5978b0d1ce591355776953f7d022206ca6569bfe841bbd: Status 404 returned error can't find the container with id 1637ded6e19336c38b5978b0d1ce591355776953f7d022206ca6569bfe841bbd
Sep 16 19:02:26.886784 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.886716    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp" event={"ID":"7c3c4403-74d8-4de4-addc-c19380e16611","Type":"ContainerStarted","Data":"9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a"}
Sep 16 19:02:26.886784 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.886763    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp" event={"ID":"7c3c4403-74d8-4de4-addc-c19380e16611","Type":"ContainerStarted","Data":"1637ded6e19336c38b5978b0d1ce591355776953f7d022206ca6569bfe841bbd"}
Sep 16 19:02:26.903898 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:26.903815    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp" podStartSLOduration=0.90380061 podStartE2EDuration="903.80061ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:26.903591482 +0000 UTC m=+1073.288150706" watchObservedRunningTime="2026-09-16 19:02:26.90380061 +0000 UTC m=+1073.288359878"
Sep 16 19:02:27.368179 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:27.368140    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:36.368257 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:36.368216    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:36.380568 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:36.380539    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:02:36.930986 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:02:36.930961    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:03:00.313112 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.313078    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"]
Sep 16 19:03:00.321094 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:03:00.321057    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:03:00.321094 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:03:00.321094 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:03:00.321094 ip-10-0-135-244 kubenswrapper[2570]:  > containerID="9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a" cmd=["/bin/sleep","15"]
Sep 16 19:03:00.321339 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.321106    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp" message="" err=<
Sep 16 19:03:00.321339 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:03:00.321339 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:03:00.321339 ip-10-0-135-244 kubenswrapper[2570]:  >
Sep 16 19:03:00.321339 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:03:00.321121    2570 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:03:00.321339 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:03:00.321339 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:03:00.321339 ip-10-0-135-244 kubenswrapper[2570]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp" podUID="7c3c4403-74d8-4de4-addc-c19380e16611" containerName="main" containerID="cri-o://9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a"
Sep 16 19:03:00.321339 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.321168    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp" podUID="7c3c4403-74d8-4de4-addc-c19380e16611" containerName="main" containerID="cri-o://9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a" gracePeriod=60
Sep 16 19:03:00.659905 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.659882    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:03:00.760650 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.760623    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-tmp-dir\") pod \"7c3c4403-74d8-4de4-addc-c19380e16611\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") "
Sep 16 19:03:00.760806 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.760653    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-model-cache\") pod \"7c3c4403-74d8-4de4-addc-c19380e16611\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") "
Sep 16 19:03:00.760806 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.760677    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-dshm\") pod \"7c3c4403-74d8-4de4-addc-c19380e16611\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") "
Sep 16 19:03:00.760806 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.760708    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-home\") pod \"7c3c4403-74d8-4de4-addc-c19380e16611\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") "
Sep 16 19:03:00.760806 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.760762    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2s4k\" (UniqueName: \"kubernetes.io/projected/7c3c4403-74d8-4de4-addc-c19380e16611-kube-api-access-c2s4k\") pod \"7c3c4403-74d8-4de4-addc-c19380e16611\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") "
Sep 16 19:03:00.760806 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.760795    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3c4403-74d8-4de4-addc-c19380e16611-tls-certs\") pod \"7c3c4403-74d8-4de4-addc-c19380e16611\" (UID: \"7c3c4403-74d8-4de4-addc-c19380e16611\") "
Sep 16 19:03:00.761056 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.760909    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c3c4403-74d8-4de4-addc-c19380e16611" (UID: "7c3c4403-74d8-4de4-addc-c19380e16611"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:00.761056 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.760931    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c3c4403-74d8-4de4-addc-c19380e16611" (UID: "7c3c4403-74d8-4de4-addc-c19380e16611"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:00.761056 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.761024    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-home" (OuterVolumeSpecName: "home") pod "7c3c4403-74d8-4de4-addc-c19380e16611" (UID: "7c3c4403-74d8-4de4-addc-c19380e16611"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:00.761184 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.761088    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:03:00.761184 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.761107    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:03:00.761184 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.761119    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:03:00.762825 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.762798    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-dshm" (OuterVolumeSpecName: "dshm") pod "7c3c4403-74d8-4de4-addc-c19380e16611" (UID: "7c3c4403-74d8-4de4-addc-c19380e16611"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:00.762825 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.762806    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3c4403-74d8-4de4-addc-c19380e16611-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c3c4403-74d8-4de4-addc-c19380e16611" (UID: "7c3c4403-74d8-4de4-addc-c19380e16611"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:00.762959 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.762935    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3c4403-74d8-4de4-addc-c19380e16611-kube-api-access-c2s4k" (OuterVolumeSpecName: "kube-api-access-c2s4k") pod "7c3c4403-74d8-4de4-addc-c19380e16611" (UID: "7c3c4403-74d8-4de4-addc-c19380e16611"). InnerVolumeSpecName "kube-api-access-c2s4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:00.861920 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.861843    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c3c4403-74d8-4de4-addc-c19380e16611-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:03:00.861920 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.861871    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2s4k\" (UniqueName: \"kubernetes.io/projected/7c3c4403-74d8-4de4-addc-c19380e16611-kube-api-access-c2s4k\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:03:00.861920 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:00.861885    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c3c4403-74d8-4de4-addc-c19380e16611-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:03:01.012141 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:01.012110    2570 generic.go:356] "Generic (PLEG): container finished" podID="7c3c4403-74d8-4de4-addc-c19380e16611" containerID="9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a" exitCode=0
Sep 16 19:03:01.012294 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:01.012158    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"
Sep 16 19:03:01.012294 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:01.012196    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp" event={"ID":"7c3c4403-74d8-4de4-addc-c19380e16611","Type":"ContainerDied","Data":"9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a"}
Sep 16 19:03:01.012294 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:01.012248    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp" event={"ID":"7c3c4403-74d8-4de4-addc-c19380e16611","Type":"ContainerDied","Data":"1637ded6e19336c38b5978b0d1ce591355776953f7d022206ca6569bfe841bbd"}
Sep 16 19:03:01.012294 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:01.012270    2570 scope.go:117] "RemoveContainer" containerID="9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a"
Sep 16 19:03:01.020569 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:01.020554    2570 scope.go:117] "RemoveContainer" containerID="9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a"
Sep 16 19:03:01.020835 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:03:01.020817    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a\": container with ID starting with 9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a not found: ID does not exist" containerID="9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a"
Sep 16 19:03:01.020885 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:01.020843    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a"} err="failed to get container status \"9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a\": rpc error: code = NotFound desc = could not find container \"9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a\": container with ID starting with 9ea12e3998830b37a3bbc80782a1cf078b34499a4c9499006f9c6da40c24ed3a not found: ID does not exist"
Sep 16 19:03:01.042340 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:01.042317    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"]
Sep 16 19:03:01.049214 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:01.045180    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fmrgsp"]
Sep 16 19:03:02.160086 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:02.160053    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c3c4403-74d8-4de4-addc-c19380e16611" path="/var/lib/kubelet/pods/7c3c4403-74d8-4de4-addc-c19380e16611/volumes"
Sep 16 19:03:55.761100 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.761064    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"]
Sep 16 19:03:55.761610 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.761444    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3c4403-74d8-4de4-addc-c19380e16611" containerName="main"
Sep 16 19:03:55.761610 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.761462    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3c4403-74d8-4de4-addc-c19380e16611" containerName="main"
Sep 16 19:03:55.761610 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.761536    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3c4403-74d8-4de4-addc-c19380e16611" containerName="main"
Sep 16 19:03:55.765603 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.765584    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.769083 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.769060    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 19:03:55.769201 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.769114    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-zcs5m\""
Sep 16 19:03:55.769201 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.769062    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 19:03:55.769201 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.769082    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 19:03:55.776136 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.776115    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"]
Sep 16 19:03:55.835335 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.835307    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7aefbb-54b3-4b22-b000-a6517d1734ac-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.835449 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.835343    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.835449 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.835363    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.835519 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.835452    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.835519 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.835490    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssfs6\" (UniqueName: \"kubernetes.io/projected/bb7aefbb-54b3-4b22-b000-a6517d1734ac-kube-api-access-ssfs6\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.835581 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.835553    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.936116 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.936090    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7aefbb-54b3-4b22-b000-a6517d1734ac-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.936255 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.936125    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.936255 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.936143    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.936255 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.936177    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.936408 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.936373    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssfs6\" (UniqueName: \"kubernetes.io/projected/bb7aefbb-54b3-4b22-b000-a6517d1734ac-kube-api-access-ssfs6\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.936463 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.936443    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.936556 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.936534    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.936720 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.936681    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.936830 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.936774    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.939106 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.939088    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.939481 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.939461    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7aefbb-54b3-4b22-b000-a6517d1734ac-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:55.944510 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:55.944487    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssfs6\" (UniqueName: \"kubernetes.io/projected/bb7aefbb-54b3-4b22-b000-a6517d1734ac-kube-api-access-ssfs6\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:56.076697 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:56.076662    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:03:56.347207 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:56.347183    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"]
Sep 16 19:03:56.349492 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:03:56.349464    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7aefbb_54b3_4b22_b000_a6517d1734ac.slice/crio-339d3bea742ee9248bf8f4374cc5d581947011ae54c06593762e23e76f88ffc6 WatchSource:0}: Error finding container 339d3bea742ee9248bf8f4374cc5d581947011ae54c06593762e23e76f88ffc6: Status 404 returned error can't find the container with id 339d3bea742ee9248bf8f4374cc5d581947011ae54c06593762e23e76f88ffc6
Sep 16 19:03:57.211356 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:57.211318    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm" event={"ID":"bb7aefbb-54b3-4b22-b000-a6517d1734ac","Type":"ContainerStarted","Data":"b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de"}
Sep 16 19:03:57.211817 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:57.211379    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm" event={"ID":"bb7aefbb-54b3-4b22-b000-a6517d1734ac","Type":"ContainerStarted","Data":"339d3bea742ee9248bf8f4374cc5d581947011ae54c06593762e23e76f88ffc6"}
Sep 16 19:03:57.233839 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:57.233788    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm" podStartSLOduration=2.233770842 podStartE2EDuration="2.233770842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:57.233407055 +0000 UTC m=+1163.617966288" watchObservedRunningTime="2026-09-16 19:03:57.233770842 +0000 UTC m=+1163.618330067"
Sep 16 19:03:58.077541 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:03:58.077509    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:04:06.077344 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:06.077310    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:04:06.089295 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:06.089268    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:04:06.255650 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:06.255623    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:04:29.836394 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:29.836353    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"]
Sep 16 19:04:29.844844 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:04:29.844800    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:04:29.844844 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:29.844844 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:29.844844 ip-10-0-135-244 kubenswrapper[2570]:  > containerID="b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de" cmd=["/bin/sleep","15"]
Sep 16 19:04:29.845088 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:29.844857    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm" message="" err=<
Sep 16 19:04:29.845088 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:29.845088 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:29.845088 ip-10-0-135-244 kubenswrapper[2570]:  >
Sep 16 19:04:29.845088 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:04:29.844869    2570 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:04:29.845088 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:29.845088 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:29.845088 ip-10-0-135-244 kubenswrapper[2570]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm" podUID="bb7aefbb-54b3-4b22-b000-a6517d1734ac" containerName="main" containerID="cri-o://b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de"
Sep 16 19:04:29.845088 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:29.844901    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm" podUID="bb7aefbb-54b3-4b22-b000-a6517d1734ac" containerName="main" containerID="cri-o://b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de" gracePeriod=60
Sep 16 19:04:30.183835 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.183812    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:04:30.326868 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.326840    2570 generic.go:356] "Generic (PLEG): container finished" podID="bb7aefbb-54b3-4b22-b000-a6517d1734ac" containerID="b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de" exitCode=0
Sep 16 19:04:30.327001 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.326878    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"
Sep 16 19:04:30.327001 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.326875    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm" event={"ID":"bb7aefbb-54b3-4b22-b000-a6517d1734ac","Type":"ContainerDied","Data":"b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de"}
Sep 16 19:04:30.327132 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.327004    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm" event={"ID":"bb7aefbb-54b3-4b22-b000-a6517d1734ac","Type":"ContainerDied","Data":"339d3bea742ee9248bf8f4374cc5d581947011ae54c06593762e23e76f88ffc6"}
Sep 16 19:04:30.327132 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.327032    2570 scope.go:117] "RemoveContainer" containerID="b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de"
Sep 16 19:04:30.334581 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.334562    2570 scope.go:117] "RemoveContainer" containerID="b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de"
Sep 16 19:04:30.334846 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:04:30.334827    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de\": container with ID starting with b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de not found: ID does not exist" containerID="b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de"
Sep 16 19:04:30.334919 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.334856    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de"} err="failed to get container status \"b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de\": rpc error: code = NotFound desc = could not find container \"b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de\": container with ID starting with b41630c837a89757aae09af5483d02ce7a3ad125172e1d60d3275006cb9ed0de not found: ID does not exist"
Sep 16 19:04:30.372119 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.372078    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-dshm\") pod \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") "
Sep 16 19:04:30.372172 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.372120    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-tmp-dir\") pod \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") "
Sep 16 19:04:30.372172 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.372148    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-model-cache\") pod \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") "
Sep 16 19:04:30.372237 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.372182    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssfs6\" (UniqueName: \"kubernetes.io/projected/bb7aefbb-54b3-4b22-b000-a6517d1734ac-kube-api-access-ssfs6\") pod \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") "
Sep 16 19:04:30.372237 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.372202    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7aefbb-54b3-4b22-b000-a6517d1734ac-tls-certs\") pod \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") "
Sep 16 19:04:30.372237 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.372228    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-home\") pod \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\" (UID: \"bb7aefbb-54b3-4b22-b000-a6517d1734ac\") "
Sep 16 19:04:30.372362 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.372336    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb7aefbb-54b3-4b22-b000-a6517d1734ac" (UID: "bb7aefbb-54b3-4b22-b000-a6517d1734ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:30.372427 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.372409    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:04:30.372530 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.372501    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-home" (OuterVolumeSpecName: "home") pod "bb7aefbb-54b3-4b22-b000-a6517d1734ac" (UID: "bb7aefbb-54b3-4b22-b000-a6517d1734ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:30.372530 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.372510    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb7aefbb-54b3-4b22-b000-a6517d1734ac" (UID: "bb7aefbb-54b3-4b22-b000-a6517d1734ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:30.374126 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.374105    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-dshm" (OuterVolumeSpecName: "dshm") pod "bb7aefbb-54b3-4b22-b000-a6517d1734ac" (UID: "bb7aefbb-54b3-4b22-b000-a6517d1734ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:30.374241 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.374221    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7aefbb-54b3-4b22-b000-a6517d1734ac-kube-api-access-ssfs6" (OuterVolumeSpecName: "kube-api-access-ssfs6") pod "bb7aefbb-54b3-4b22-b000-a6517d1734ac" (UID: "bb7aefbb-54b3-4b22-b000-a6517d1734ac"). InnerVolumeSpecName "kube-api-access-ssfs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:30.374297 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.374235    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb7aefbb-54b3-4b22-b000-a6517d1734ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb7aefbb-54b3-4b22-b000-a6517d1734ac" (UID: "bb7aefbb-54b3-4b22-b000-a6517d1734ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:30.473607 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.473586    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:04:30.473607 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.473606    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssfs6\" (UniqueName: \"kubernetes.io/projected/bb7aefbb-54b3-4b22-b000-a6517d1734ac-kube-api-access-ssfs6\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:04:30.473732 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.473616    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7aefbb-54b3-4b22-b000-a6517d1734ac-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:04:30.473732 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.473624    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:04:30.473732 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.473632    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7aefbb-54b3-4b22-b000-a6517d1734ac-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:04:30.649969 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.649928    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"]
Sep 16 19:04:30.657927 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:30.657906    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-vs2dm"]
Sep 16 19:04:32.160030 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:04:32.160000    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb7aefbb-54b3-4b22-b000-a6517d1734ac" path="/var/lib/kubelet/pods/bb7aefbb-54b3-4b22-b000-a6517d1734ac/volumes"
Sep 16 19:05:07.883673 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:07.883642    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"]
Sep 16 19:05:07.884184 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:07.883900    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb7aefbb-54b3-4b22-b000-a6517d1734ac" containerName="main"
Sep 16 19:05:07.884184 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:07.883913    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb7aefbb-54b3-4b22-b000-a6517d1734ac" containerName="main"
Sep 16 19:05:07.884184 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:07.883962    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb7aefbb-54b3-4b22-b000-a6517d1734ac" containerName="main"
Sep 16 19:05:07.886838 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:07.886816    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"
Sep 16 19:05:07.897845 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:07.897819    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 19:05:07.897999 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:07.897819    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 19:05:07.898090 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:07.897835    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-pdp7f\""
Sep 16 19:05:07.898154 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:07.897875    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-tbw9w\""
Sep 16 19:05:07.898154 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:07.897935    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 19:05:07.900195 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:07.900170    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"]
Sep 16 19:05:07.916564 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:07.916543    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k55s7\" (UniqueName: \"kubernetes.io/projected/8e81aee2-7d86-4961-aac2-9a6d0f1a199c-kube-api-access-k55s7\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj\" (UID: \"8e81aee2-7d86-4961-aac2-9a6d0f1a199c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"
Sep 16 19:05:07.916667 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:07.916601    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e81aee2-7d86-4961-aac2-9a6d0f1a199c-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj\" (UID: \"8e81aee2-7d86-4961-aac2-9a6d0f1a199c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"
Sep 16 19:05:08.017865 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:08.017835    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e81aee2-7d86-4961-aac2-9a6d0f1a199c-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj\" (UID: \"8e81aee2-7d86-4961-aac2-9a6d0f1a199c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"
Sep 16 19:05:08.017989 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:08.017898    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k55s7\" (UniqueName: \"kubernetes.io/projected/8e81aee2-7d86-4961-aac2-9a6d0f1a199c-kube-api-access-k55s7\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj\" (UID: \"8e81aee2-7d86-4961-aac2-9a6d0f1a199c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"
Sep 16 19:05:08.020265 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:08.020236    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e81aee2-7d86-4961-aac2-9a6d0f1a199c-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj\" (UID: \"8e81aee2-7d86-4961-aac2-9a6d0f1a199c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"
Sep 16 19:05:08.029567 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:08.029541    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k55s7\" (UniqueName: \"kubernetes.io/projected/8e81aee2-7d86-4961-aac2-9a6d0f1a199c-kube-api-access-k55s7\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj\" (UID: \"8e81aee2-7d86-4961-aac2-9a6d0f1a199c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"
Sep 16 19:05:08.198564 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:08.198501    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"
Sep 16 19:05:08.467969 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:08.467943    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"]
Sep 16 19:05:08.470129 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:05:08.470103    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e81aee2_7d86_4961_aac2_9a6d0f1a199c.slice/crio-8d6c9fc3853c901fc045204bb6bacf7ad6c8b863bc0e6b3b3ad0e70d85aeb6e3 WatchSource:0}: Error finding container 8d6c9fc3853c901fc045204bb6bacf7ad6c8b863bc0e6b3b3ad0e70d85aeb6e3: Status 404 returned error can't find the container with id 8d6c9fc3853c901fc045204bb6bacf7ad6c8b863bc0e6b3b3ad0e70d85aeb6e3
Sep 16 19:05:08.472714 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:08.472674    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:05:09.461958 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:09.461915    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj" event={"ID":"8e81aee2-7d86-4961-aac2-9a6d0f1a199c","Type":"ContainerStarted","Data":"8d6c9fc3853c901fc045204bb6bacf7ad6c8b863bc0e6b3b3ad0e70d85aeb6e3"}
Sep 16 19:05:11.470756 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:11.470719    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj" event={"ID":"8e81aee2-7d86-4961-aac2-9a6d0f1a199c","Type":"ContainerStarted","Data":"59fc12a7b871c5210c041cca1b3d554dc698d8e62d3857b95a4e834d3fcee554"}
Sep 16 19:05:11.471112 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:11.470786    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"
Sep 16 19:05:11.489933 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:11.489881    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj" podStartSLOduration=2.274166285 podStartE2EDuration="4.489866317s" totalImagesPullingTime="2.215700032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:11.488491113 +0000 UTC m=+1237.873050337" watchObservedRunningTime="2026-09-16 19:05:11.489866317 +0000 UTC m=+1237.874425540"
Sep 16 19:05:42.477317 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:05:42.477289    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"
Sep 16 19:07:10.835159 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:10.835123    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"]
Sep 16 19:07:10.837010 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:10.836992    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:10.840864 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:10.840842    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 16 19:07:10.841955 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:10.841924    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-2xzts\""
Sep 16 19:07:10.842089 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:10.841957    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 16 19:07:10.842089 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:10.841956    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 16 19:07:10.842089 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:10.841924    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-tm6gs\""
Sep 16 19:07:10.864727 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:10.864702    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"]
Sep 16 19:07:10.978604 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:10.978580    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:10.978743 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:10.978624    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:10.978743 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:10.978647    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5b5t\" (UniqueName: \"kubernetes.io/projected/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-kube-api-access-w5b5t\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:10.978743 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:10.978715    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:11.079898 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:11.079867    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:11.080006 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:11.079903    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5b5t\" (UniqueName: \"kubernetes.io/projected/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-kube-api-access-w5b5t\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:11.080006 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:11.079938    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:11.080092 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:11.080003    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:11.080318 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:11.080300    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:11.080318 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:11.080312    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:11.082327 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:11.082307    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:11.090176 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:11.090119    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5b5t\" (UniqueName: \"kubernetes.io/projected/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-kube-api-access-w5b5t\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:11.147085 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:11.147064    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:11.418902 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:11.418835    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"]
Sep 16 19:07:11.422642 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:07:11.422617    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f561d3_7f5a_4a7a_93f4_7bc1f410e6c3.slice/crio-85d8639265f30f701f8943a4a0efd476b081db4fe1c5b15ed254b63d0c986473 WatchSource:0}: Error finding container 85d8639265f30f701f8943a4a0efd476b081db4fe1c5b15ed254b63d0c986473: Status 404 returned error can't find the container with id 85d8639265f30f701f8943a4a0efd476b081db4fe1c5b15ed254b63d0c986473
Sep 16 19:07:11.900883 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:11.900852    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf" event={"ID":"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3","Type":"ContainerStarted","Data":"843eea6040d2b72d1c49a7f252716ba5805237e117b6c29f749d38f211d5c8b7"}
Sep 16 19:07:11.901263 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:11.900900    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf" event={"ID":"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3","Type":"ContainerStarted","Data":"85d8639265f30f701f8943a4a0efd476b081db4fe1c5b15ed254b63d0c986473"}
Sep 16 19:07:11.901263 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:11.900952    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:11.919955 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:11.919907    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf" podStartSLOduration=1.91989091 podStartE2EDuration="1.91989091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:11.917480348 +0000 UTC m=+1358.302039573" watchObservedRunningTime="2026-09-16 19:07:11.91989091 +0000 UTC m=+1358.304450140"
Sep 16 19:07:22.183728 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:22.183662    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"]
Sep 16 19:07:22.476999 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:22.476916    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj" podUID="8e81aee2-7d86-4961-aac2-9a6d0f1a199c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:07:32.477138 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:32.477098    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj" podUID="8e81aee2-7d86-4961-aac2-9a6d0f1a199c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:07:37.202857 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:37.202798    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj" podUID="8e81aee2-7d86-4961-aac2-9a6d0f1a199c" containerName="main" containerID="cri-o://59fc12a7b871c5210c041cca1b3d554dc698d8e62d3857b95a4e834d3fcee554" gracePeriod=45
Sep 16 19:07:42.477415 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:42.477381    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj" podUID="8e81aee2-7d86-4961-aac2-9a6d0f1a199c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:07:42.477807 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:42.477478    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"
Sep 16 19:07:42.905556 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:42.905529    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:07:45.128361 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:45.128319    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"]
Sep 16 19:07:52.477259 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:52.477217    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj" podUID="8e81aee2-7d86-4961-aac2-9a6d0f1a199c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:07:52.906033 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:07:52.905995    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf" podUID="67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:00.147356 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:00.147316    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf" podUID="67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3" containerName="main" containerID="cri-o://843eea6040d2b72d1c49a7f252716ba5805237e117b6c29f749d38f211d5c8b7" gracePeriod=45
Sep 16 19:08:02.477131 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:02.477095    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj" podUID="8e81aee2-7d86-4961-aac2-9a6d0f1a199c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:02.905480 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:02.905447    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf" podUID="67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:07.434579 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:07.434557    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"
Sep 16 19:08:07.468834 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:07.468767    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e81aee2-7d86-4961-aac2-9a6d0f1a199c-tls-certs\") pod \"8e81aee2-7d86-4961-aac2-9a6d0f1a199c\" (UID: \"8e81aee2-7d86-4961-aac2-9a6d0f1a199c\") "
Sep 16 19:08:07.468965 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:07.468831    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k55s7\" (UniqueName: \"kubernetes.io/projected/8e81aee2-7d86-4961-aac2-9a6d0f1a199c-kube-api-access-k55s7\") pod \"8e81aee2-7d86-4961-aac2-9a6d0f1a199c\" (UID: \"8e81aee2-7d86-4961-aac2-9a6d0f1a199c\") "
Sep 16 19:08:07.470932 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:07.470906    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e81aee2-7d86-4961-aac2-9a6d0f1a199c-kube-api-access-k55s7" (OuterVolumeSpecName: "kube-api-access-k55s7") pod "8e81aee2-7d86-4961-aac2-9a6d0f1a199c" (UID: "8e81aee2-7d86-4961-aac2-9a6d0f1a199c"). InnerVolumeSpecName "kube-api-access-k55s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:07.471025 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:07.470967    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e81aee2-7d86-4961-aac2-9a6d0f1a199c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e81aee2-7d86-4961-aac2-9a6d0f1a199c" (UID: "8e81aee2-7d86-4961-aac2-9a6d0f1a199c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:07.569386 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:07.569368    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k55s7\" (UniqueName: \"kubernetes.io/projected/8e81aee2-7d86-4961-aac2-9a6d0f1a199c-kube-api-access-k55s7\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:08:07.569386 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:07.569387    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e81aee2-7d86-4961-aac2-9a6d0f1a199c-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:08:08.106931 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:08.106902    2570 generic.go:356] "Generic (PLEG): container finished" podID="8e81aee2-7d86-4961-aac2-9a6d0f1a199c" containerID="59fc12a7b871c5210c041cca1b3d554dc698d8e62d3857b95a4e834d3fcee554" exitCode=0
Sep 16 19:08:08.107084 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:08.106955    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"
Sep 16 19:08:08.107084 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:08.106983    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj" event={"ID":"8e81aee2-7d86-4961-aac2-9a6d0f1a199c","Type":"ContainerDied","Data":"59fc12a7b871c5210c041cca1b3d554dc698d8e62d3857b95a4e834d3fcee554"}
Sep 16 19:08:08.107084 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:08.107032    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj" event={"ID":"8e81aee2-7d86-4961-aac2-9a6d0f1a199c","Type":"ContainerDied","Data":"8d6c9fc3853c901fc045204bb6bacf7ad6c8b863bc0e6b3b3ad0e70d85aeb6e3"}
Sep 16 19:08:08.107084 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:08.107054    2570 scope.go:117] "RemoveContainer" containerID="59fc12a7b871c5210c041cca1b3d554dc698d8e62d3857b95a4e834d3fcee554"
Sep 16 19:08:08.115192 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:08.115174    2570 scope.go:117] "RemoveContainer" containerID="59fc12a7b871c5210c041cca1b3d554dc698d8e62d3857b95a4e834d3fcee554"
Sep 16 19:08:08.115473 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:08:08.115455    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59fc12a7b871c5210c041cca1b3d554dc698d8e62d3857b95a4e834d3fcee554\": container with ID starting with 59fc12a7b871c5210c041cca1b3d554dc698d8e62d3857b95a4e834d3fcee554 not found: ID does not exist" containerID="59fc12a7b871c5210c041cca1b3d554dc698d8e62d3857b95a4e834d3fcee554"
Sep 16 19:08:08.115512 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:08.115481    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59fc12a7b871c5210c041cca1b3d554dc698d8e62d3857b95a4e834d3fcee554"} err="failed to get container status \"59fc12a7b871c5210c041cca1b3d554dc698d8e62d3857b95a4e834d3fcee554\": rpc error: code = NotFound desc = could not find container \"59fc12a7b871c5210c041cca1b3d554dc698d8e62d3857b95a4e834d3fcee554\": container with ID starting with 59fc12a7b871c5210c041cca1b3d554dc698d8e62d3857b95a4e834d3fcee554 not found: ID does not exist"
Sep 16 19:08:08.127041 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:08.127019    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"]
Sep 16 19:08:08.135697 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:08.135664    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cnlnrj"]
Sep 16 19:08:08.159814 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:08.159784    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e81aee2-7d86-4961-aac2-9a6d0f1a199c" path="/var/lib/kubelet/pods/8e81aee2-7d86-4961-aac2-9a6d0f1a199c/volumes"
Sep 16 19:08:12.906275 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:12.906235    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf" podUID="67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:12.906752 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:12.906347    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:08:22.905595 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:22.905560    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf" podUID="67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:30.385046 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:30.385023    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:08:30.422391 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:30.422340    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5b5t\" (UniqueName: \"kubernetes.io/projected/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-kube-api-access-w5b5t\") pod \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") "
Sep 16 19:08:30.422484 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:30.422401    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-tokenizer-cache\") pod \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") "
Sep 16 19:08:30.422484 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:30.422428    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-cachi2-cache\") pod \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") "
Sep 16 19:08:30.422610 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:30.422587    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-tls-certs\") pod \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\" (UID: \"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3\") "
Sep 16 19:08:30.422673 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:30.422631    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3" (UID: "67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:30.422752 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:30.422699    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3" (UID: "67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:30.422857 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:30.422840    2570 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-tokenizer-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:08:30.422897 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:30.422859    2570 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-cachi2-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:08:30.424536 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:30.424510    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3" (UID: "67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:30.424700 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:30.424672    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-kube-api-access-w5b5t" (OuterVolumeSpecName: "kube-api-access-w5b5t") pod "67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3" (UID: "67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3"). InnerVolumeSpecName "kube-api-access-w5b5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:30.523964 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:30.523938    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5b5t\" (UniqueName: \"kubernetes.io/projected/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-kube-api-access-w5b5t\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:08:30.523964 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:30.523960    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:08:31.191930 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:31.191888    2570 generic.go:356] "Generic (PLEG): container finished" podID="67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3" containerID="843eea6040d2b72d1c49a7f252716ba5805237e117b6c29f749d38f211d5c8b7" exitCode=0
Sep 16 19:08:31.192117 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:31.191936    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"
Sep 16 19:08:31.192117 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:31.191969    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf" event={"ID":"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3","Type":"ContainerDied","Data":"843eea6040d2b72d1c49a7f252716ba5805237e117b6c29f749d38f211d5c8b7"}
Sep 16 19:08:31.192117 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:31.192010    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf" event={"ID":"67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3","Type":"ContainerDied","Data":"85d8639265f30f701f8943a4a0efd476b081db4fe1c5b15ed254b63d0c986473"}
Sep 16 19:08:31.192117 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:31.192035    2570 scope.go:117] "RemoveContainer" containerID="843eea6040d2b72d1c49a7f252716ba5805237e117b6c29f749d38f211d5c8b7"
Sep 16 19:08:31.200379 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:31.200363    2570 scope.go:117] "RemoveContainer" containerID="843eea6040d2b72d1c49a7f252716ba5805237e117b6c29f749d38f211d5c8b7"
Sep 16 19:08:31.200641 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:08:31.200623    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843eea6040d2b72d1c49a7f252716ba5805237e117b6c29f749d38f211d5c8b7\": container with ID starting with 843eea6040d2b72d1c49a7f252716ba5805237e117b6c29f749d38f211d5c8b7 not found: ID does not exist" containerID="843eea6040d2b72d1c49a7f252716ba5805237e117b6c29f749d38f211d5c8b7"
Sep 16 19:08:31.200743 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:31.200653    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843eea6040d2b72d1c49a7f252716ba5805237e117b6c29f749d38f211d5c8b7"} err="failed to get container status \"843eea6040d2b72d1c49a7f252716ba5805237e117b6c29f749d38f211d5c8b7\": rpc error: code = NotFound desc = could not find container \"843eea6040d2b72d1c49a7f252716ba5805237e117b6c29f749d38f211d5c8b7\": container with ID starting with 843eea6040d2b72d1c49a7f252716ba5805237e117b6c29f749d38f211d5c8b7 not found: ID does not exist"
Sep 16 19:08:31.214312 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:31.214291    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"]
Sep 16 19:08:31.218450 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:31.218429    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcft2vf"]
Sep 16 19:08:32.159815 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:32.159786    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3" path="/var/lib/kubelet/pods/67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3/volumes"
Sep 16 19:08:38.856044 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.856012    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"]
Sep 16 19:08:38.856390 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.856245    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3" containerName="main"
Sep 16 19:08:38.856390 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.856256    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3" containerName="main"
Sep 16 19:08:38.856390 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.856264    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e81aee2-7d86-4961-aac2-9a6d0f1a199c" containerName="main"
Sep 16 19:08:38.856390 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.856269    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e81aee2-7d86-4961-aac2-9a6d0f1a199c" containerName="main"
Sep 16 19:08:38.856390 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.856310    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e81aee2-7d86-4961-aac2-9a6d0f1a199c" containerName="main"
Sep 16 19:08:38.856390 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.856318    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f561d3-7f5a-4a7a-93f4-7bc1f410e6c3" containerName="main"
Sep 16 19:08:38.859009 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.858992    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:38.861969 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.861933    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 19:08:38.862118 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.862034    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 19:08:38.862895 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.862872    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-72vmh\""
Sep 16 19:08:38.863054 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.863035    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-q7pv7\""
Sep 16 19:08:38.863145 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.863086    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 19:08:38.863458 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.863441    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"]
Sep 16 19:08:38.866407 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.866385    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:38.872456 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.872434    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"]
Sep 16 19:08:38.876971 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.876950    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"]
Sep 16 19:08:38.981612 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.981591    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dpmb\" (UniqueName: \"kubernetes.io/projected/98d26def-6529-44b4-81f1-2495a5a88631-kube-api-access-7dpmb\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:38.981738 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.981619    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:38.981738 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.981642    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:38.981738 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.981666    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:38.981840 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.981744    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98d26def-6529-44b4-81f1-2495a5a88631-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:38.981840 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.981773    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:38.981840 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.981790    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd761dbe-208f-436b-b9d5-47b40ec62b05-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:38.981840 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.981808    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:38.981840 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.981829    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:38.981984 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.981897    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:38.981984 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.981930    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4p5j\" (UniqueName: \"kubernetes.io/projected/dd761dbe-208f-436b-b9d5-47b40ec62b05-kube-api-access-n4p5j\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:38.981984 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:38.981957    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:39.082873 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.082851    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:39.082966 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.082880    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:39.082966 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.082899    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:39.082966 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.082917    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4p5j\" (UniqueName: \"kubernetes.io/projected/dd761dbe-208f-436b-b9d5-47b40ec62b05-kube-api-access-n4p5j\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:39.083080 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.082985    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:39.083080 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.083039    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dpmb\" (UniqueName: \"kubernetes.io/projected/98d26def-6529-44b4-81f1-2495a5a88631-kube-api-access-7dpmb\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:39.083080 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.083065    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:39.083242 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.083100    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:39.083242 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.083149    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:39.083242 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.083188    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98d26def-6529-44b4-81f1-2495a5a88631-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:39.083242 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.083218    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:39.083460 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.083242    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd761dbe-208f-436b-b9d5-47b40ec62b05-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:39.083460 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.083302    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:39.083460 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.083312    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:39.083460 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.083304    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:39.083460 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.083438    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:39.083460 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.083438    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:39.083761 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.083627    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:39.085718 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.085674    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:39.085800 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.085722    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98d26def-6529-44b4-81f1-2495a5a88631-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:39.085800 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.085767    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:39.086169 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.086147    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd761dbe-208f-436b-b9d5-47b40ec62b05-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:39.092674 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.092642    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4p5j\" (UniqueName: \"kubernetes.io/projected/dd761dbe-208f-436b-b9d5-47b40ec62b05-kube-api-access-n4p5j\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:39.095351 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.095323    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dpmb\" (UniqueName: \"kubernetes.io/projected/98d26def-6529-44b4-81f1-2495a5a88631-kube-api-access-7dpmb\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:39.169206 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.169147    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:39.177997 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.177979    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:39.454760 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.454738    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"]
Sep 16 19:08:39.457441 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:39.457418    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"]
Sep 16 19:08:39.457545 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:08:39.457484    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd761dbe_208f_436b_b9d5_47b40ec62b05.slice/crio-3219c309e68a6581edc7e0650a9abd00c535bb3f84d7d6941b1779471fb5204d WatchSource:0}: Error finding container 3219c309e68a6581edc7e0650a9abd00c535bb3f84d7d6941b1779471fb5204d: Status 404 returned error can't find the container with id 3219c309e68a6581edc7e0650a9abd00c535bb3f84d7d6941b1779471fb5204d
Sep 16 19:08:39.459460 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:08:39.459436    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d26def_6529_44b4_81f1_2495a5a88631.slice/crio-51352cde3a6f0b89e9d547a6c90060d8a6fa522fd7103c5cfa2d180a413ba7c1 WatchSource:0}: Error finding container 51352cde3a6f0b89e9d547a6c90060d8a6fa522fd7103c5cfa2d180a413ba7c1: Status 404 returned error can't find the container with id 51352cde3a6f0b89e9d547a6c90060d8a6fa522fd7103c5cfa2d180a413ba7c1
Sep 16 19:08:40.226466 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:40.226432    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7" event={"ID":"98d26def-6529-44b4-81f1-2495a5a88631","Type":"ContainerStarted","Data":"c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727"}
Sep 16 19:08:40.226928 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:40.226494    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7" event={"ID":"98d26def-6529-44b4-81f1-2495a5a88631","Type":"ContainerStarted","Data":"51352cde3a6f0b89e9d547a6c90060d8a6fa522fd7103c5cfa2d180a413ba7c1"}
Sep 16 19:08:40.227909 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:40.227874    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6" event={"ID":"dd761dbe-208f-436b-b9d5-47b40ec62b05","Type":"ContainerStarted","Data":"3219c309e68a6581edc7e0650a9abd00c535bb3f84d7d6941b1779471fb5204d"}
Sep 16 19:08:40.249292 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:40.249244    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7" podStartSLOduration=2.249222811 podStartE2EDuration="2.249222811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:40.247082497 +0000 UTC m=+1446.631641720" watchObservedRunningTime="2026-09-16 19:08:40.249222811 +0000 UTC m=+1446.633782034"
Sep 16 19:08:41.178403 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:41.178375    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:41.233583 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:41.233553    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6" event={"ID":"dd761dbe-208f-436b-b9d5-47b40ec62b05","Type":"ContainerStarted","Data":"1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5"}
Sep 16 19:08:41.233953 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:41.233733    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:42.241644 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:42.241611    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6" event={"ID":"dd761dbe-208f-436b-b9d5-47b40ec62b05","Type":"ContainerStarted","Data":"0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a"}
Sep 16 19:08:42.262991 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:42.262936    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6" podStartSLOduration=3.036049677 podStartE2EDuration="4.26291718s" totalImagesPullingTime="1.226867503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:42.261774644 +0000 UTC m=+1448.646333867" watchObservedRunningTime="2026-09-16 19:08:42.26291718 +0000 UTC m=+1448.647476404"
Sep 16 19:08:43.170142 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:43.170111    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:49.170051 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:49.170008    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:49.178341 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:49.178313    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:49.182335 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:49.182311    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:49.190093 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:49.190072    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:49.278123 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:49.278099    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:08:49.279002 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:49.278983    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:08:50.282767 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:08:50.282732    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:09:19.419546 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:19.419512    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"]
Sep 16 19:09:19.422950 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:19.422918    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"]
Sep 16 19:09:19.428764 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:09:19.428732    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:09:19.428764 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:19.428764 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:19.428764 ip-10-0-135-244 kubenswrapper[2570]:  > containerID="c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727" cmd=["/bin/sleep","15"]
Sep 16 19:09:19.429062 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:19.428780    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7" message="" err=<
Sep 16 19:09:19.429062 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:19.429062 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:19.429062 ip-10-0-135-244 kubenswrapper[2570]:  >
Sep 16 19:09:19.429062 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:09:19.428792    2570 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:09:19.429062 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:19.429062 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:19.429062 ip-10-0-135-244 kubenswrapper[2570]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7" podUID="98d26def-6529-44b4-81f1-2495a5a88631" containerName="main" containerID="cri-o://c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727"
Sep 16 19:09:19.429062 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:19.428825    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7" podUID="98d26def-6529-44b4-81f1-2495a5a88631" containerName="main" containerID="cri-o://c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727" gracePeriod=60
Sep 16 19:09:19.433404 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:09:19.433373    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:09:19.433404 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:19.433404 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:19.433404 ip-10-0-135-244 kubenswrapper[2570]:  > containerID="0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a" cmd=["/bin/sleep","15"]
Sep 16 19:09:19.433647 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:19.433419    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6" message="" err=<
Sep 16 19:09:19.433647 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:19.433647 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:19.433647 ip-10-0-135-244 kubenswrapper[2570]:  >
Sep 16 19:09:19.433647 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:09:19.433435    2570 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:09:19.433647 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:19.433647 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:19.433647 ip-10-0-135-244 kubenswrapper[2570]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6" podUID="dd761dbe-208f-436b-b9d5-47b40ec62b05" containerName="main" containerID="cri-o://0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a"
Sep 16 19:09:19.433647 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:19.433472    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6" podUID="dd761dbe-208f-436b-b9d5-47b40ec62b05" containerName="main" containerID="cri-o://0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a" gracePeriod=60
Sep 16 19:09:20.045491 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.045442    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6" podUID="dd761dbe-208f-436b-b9d5-47b40ec62b05" containerName="llm-d-routing-sidecar" containerID="cri-o://1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5" gracePeriod=60
Sep 16 19:09:20.169581 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.169558    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:09:20.272225 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.272197    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-dshm\") pod \"98d26def-6529-44b4-81f1-2495a5a88631\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") "
Sep 16 19:09:20.272360 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.272242    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dpmb\" (UniqueName: \"kubernetes.io/projected/98d26def-6529-44b4-81f1-2495a5a88631-kube-api-access-7dpmb\") pod \"98d26def-6529-44b4-81f1-2495a5a88631\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") "
Sep 16 19:09:20.272360 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.272309    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-tmp-dir\") pod \"98d26def-6529-44b4-81f1-2495a5a88631\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") "
Sep 16 19:09:20.272360 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.272331    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98d26def-6529-44b4-81f1-2495a5a88631-tls-certs\") pod \"98d26def-6529-44b4-81f1-2495a5a88631\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") "
Sep 16 19:09:20.272509 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.272366    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-model-cache\") pod \"98d26def-6529-44b4-81f1-2495a5a88631\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") "
Sep 16 19:09:20.272509 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.272402    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-home\") pod \"98d26def-6529-44b4-81f1-2495a5a88631\" (UID: \"98d26def-6529-44b4-81f1-2495a5a88631\") "
Sep 16 19:09:20.272649 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.272581    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98d26def-6529-44b4-81f1-2495a5a88631" (UID: "98d26def-6529-44b4-81f1-2495a5a88631"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:20.273038 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.272926    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:09:20.273210 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.273064    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-home" (OuterVolumeSpecName: "home") pod "98d26def-6529-44b4-81f1-2495a5a88631" (UID: "98d26def-6529-44b4-81f1-2495a5a88631"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:20.273210 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.273106    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-model-cache" (OuterVolumeSpecName: "model-cache") pod "98d26def-6529-44b4-81f1-2495a5a88631" (UID: "98d26def-6529-44b4-81f1-2495a5a88631"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:20.274465 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.274439    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-dshm" (OuterVolumeSpecName: "dshm") pod "98d26def-6529-44b4-81f1-2495a5a88631" (UID: "98d26def-6529-44b4-81f1-2495a5a88631"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:20.274633 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.274571    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d26def-6529-44b4-81f1-2495a5a88631-kube-api-access-7dpmb" (OuterVolumeSpecName: "kube-api-access-7dpmb") pod "98d26def-6529-44b4-81f1-2495a5a88631" (UID: "98d26def-6529-44b4-81f1-2495a5a88631"). InnerVolumeSpecName "kube-api-access-7dpmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:20.274838 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.274821    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98d26def-6529-44b4-81f1-2495a5a88631-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98d26def-6529-44b4-81f1-2495a5a88631" (UID: "98d26def-6529-44b4-81f1-2495a5a88631"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:20.284323 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.284304    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:09:20.373186 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.373116    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-home\") pod \"dd761dbe-208f-436b-b9d5-47b40ec62b05\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") "
Sep 16 19:09:20.373186 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.373163    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-tmp-dir\") pod \"dd761dbe-208f-436b-b9d5-47b40ec62b05\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") "
Sep 16 19:09:20.373377 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.373207    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd761dbe-208f-436b-b9d5-47b40ec62b05-tls-certs\") pod \"dd761dbe-208f-436b-b9d5-47b40ec62b05\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") "
Sep 16 19:09:20.373377 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.373235    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-model-cache\") pod \"dd761dbe-208f-436b-b9d5-47b40ec62b05\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") "
Sep 16 19:09:20.373377 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.373272    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-dshm\") pod \"dd761dbe-208f-436b-b9d5-47b40ec62b05\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") "
Sep 16 19:09:20.373377 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.373330    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4p5j\" (UniqueName: \"kubernetes.io/projected/dd761dbe-208f-436b-b9d5-47b40ec62b05-kube-api-access-n4p5j\") pod \"dd761dbe-208f-436b-b9d5-47b40ec62b05\" (UID: \"dd761dbe-208f-436b-b9d5-47b40ec62b05\") "
Sep 16 19:09:20.373560 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.373378    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-home" (OuterVolumeSpecName: "home") pod "dd761dbe-208f-436b-b9d5-47b40ec62b05" (UID: "dd761dbe-208f-436b-b9d5-47b40ec62b05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:20.373560 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.373490    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd761dbe-208f-436b-b9d5-47b40ec62b05" (UID: "dd761dbe-208f-436b-b9d5-47b40ec62b05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:20.373993 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.373519    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd761dbe-208f-436b-b9d5-47b40ec62b05" (UID: "dd761dbe-208f-436b-b9d5-47b40ec62b05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:20.373993 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.373972    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dpmb\" (UniqueName: \"kubernetes.io/projected/98d26def-6529-44b4-81f1-2495a5a88631-kube-api-access-7dpmb\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:09:20.374174 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.374008    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98d26def-6529-44b4-81f1-2495a5a88631-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:09:20.374174 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.374023    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:09:20.374174 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.374042    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:09:20.374174 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.374055    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:09:20.374174 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.374071    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98d26def-6529-44b4-81f1-2495a5a88631-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:09:20.374174 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.374086    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:09:20.376224 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.376193    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd761dbe-208f-436b-b9d5-47b40ec62b05-kube-api-access-n4p5j" (OuterVolumeSpecName: "kube-api-access-n4p5j") pod "dd761dbe-208f-436b-b9d5-47b40ec62b05" (UID: "dd761dbe-208f-436b-b9d5-47b40ec62b05"). InnerVolumeSpecName "kube-api-access-n4p5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:20.377204 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.376424    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-dshm" (OuterVolumeSpecName: "dshm") pod "dd761dbe-208f-436b-b9d5-47b40ec62b05" (UID: "dd761dbe-208f-436b-b9d5-47b40ec62b05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:20.378995 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.378066    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd761dbe-208f-436b-b9d5-47b40ec62b05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd761dbe-208f-436b-b9d5-47b40ec62b05" (UID: "dd761dbe-208f-436b-b9d5-47b40ec62b05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:20.383888 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.383859    2570 generic.go:356] "Generic (PLEG): container finished" podID="dd761dbe-208f-436b-b9d5-47b40ec62b05" containerID="0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a" exitCode=0
Sep 16 19:09:20.383888 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.383886    2570 generic.go:356] "Generic (PLEG): container finished" podID="dd761dbe-208f-436b-b9d5-47b40ec62b05" containerID="1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5" exitCode=0
Sep 16 19:09:20.384039 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.383917    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"
Sep 16 19:09:20.384039 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.383946    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6" event={"ID":"dd761dbe-208f-436b-b9d5-47b40ec62b05","Type":"ContainerDied","Data":"0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a"}
Sep 16 19:09:20.384039 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.384027    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6" event={"ID":"dd761dbe-208f-436b-b9d5-47b40ec62b05","Type":"ContainerDied","Data":"1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5"}
Sep 16 19:09:20.384197 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.384057    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6" event={"ID":"dd761dbe-208f-436b-b9d5-47b40ec62b05","Type":"ContainerDied","Data":"3219c309e68a6581edc7e0650a9abd00c535bb3f84d7d6941b1779471fb5204d"}
Sep 16 19:09:20.384197 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.384084    2570 scope.go:117] "RemoveContainer" containerID="0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a"
Sep 16 19:09:20.385530 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.385511    2570 generic.go:356] "Generic (PLEG): container finished" podID="98d26def-6529-44b4-81f1-2495a5a88631" containerID="c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727" exitCode=0
Sep 16 19:09:20.385637 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.385548    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"
Sep 16 19:09:20.385637 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.385574    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7" event={"ID":"98d26def-6529-44b4-81f1-2495a5a88631","Type":"ContainerDied","Data":"c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727"}
Sep 16 19:09:20.385637 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.385601    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7" event={"ID":"98d26def-6529-44b4-81f1-2495a5a88631","Type":"ContainerDied","Data":"51352cde3a6f0b89e9d547a6c90060d8a6fa522fd7103c5cfa2d180a413ba7c1"}
Sep 16 19:09:20.392350 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.392328    2570 scope.go:117] "RemoveContainer" containerID="1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5"
Sep 16 19:09:20.399444 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.399428    2570 scope.go:117] "RemoveContainer" containerID="0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a"
Sep 16 19:09:20.399797 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:09:20.399682    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a\": container with ID starting with 0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a not found: ID does not exist" containerID="0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a"
Sep 16 19:09:20.399843 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.399805    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a"} err="failed to get container status \"0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a\": rpc error: code = NotFound desc = could not find container \"0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a\": container with ID starting with 0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a not found: ID does not exist"
Sep 16 19:09:20.399843 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.399822    2570 scope.go:117] "RemoveContainer" containerID="1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5"
Sep 16 19:09:20.400068 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:09:20.400051    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5\": container with ID starting with 1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5 not found: ID does not exist" containerID="1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5"
Sep 16 19:09:20.400115 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.400074    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5"} err="failed to get container status \"1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5\": rpc error: code = NotFound desc = could not find container \"1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5\": container with ID starting with 1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5 not found: ID does not exist"
Sep 16 19:09:20.400115 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.400089    2570 scope.go:117] "RemoveContainer" containerID="0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a"
Sep 16 19:09:20.400289 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.400273    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a"} err="failed to get container status \"0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a\": rpc error: code = NotFound desc = could not find container \"0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a\": container with ID starting with 0914646b4aa4ef5775bea9fd9ff5e0c0afc2329835756e6134aae8c5e1d5953a not found: ID does not exist"
Sep 16 19:09:20.400326 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.400289    2570 scope.go:117] "RemoveContainer" containerID="1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5"
Sep 16 19:09:20.400467 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.400454    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5"} err="failed to get container status \"1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5\": rpc error: code = NotFound desc = could not find container \"1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5\": container with ID starting with 1f4f01a05a4c93c186b9bafa90843b225af43b41ddfbfa77d7386480a67663d5 not found: ID does not exist"
Sep 16 19:09:20.400503 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.400467    2570 scope.go:117] "RemoveContainer" containerID="c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727"
Sep 16 19:09:20.407765 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.407745    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"]
Sep 16 19:09:20.408078 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.408060    2570 scope.go:117] "RemoveContainer" containerID="c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727"
Sep 16 19:09:20.408340 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:09:20.408322    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727\": container with ID starting with c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727 not found: ID does not exist" containerID="c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727"
Sep 16 19:09:20.408392 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.408345    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727"} err="failed to get container status \"c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727\": rpc error: code = NotFound desc = could not find container \"c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727\": container with ID starting with c284d36e80a80557f919bf30ed09bb44eada20b4a28e3f0be912441b1aa36727 not found: ID does not exist"
Sep 16 19:09:20.411838 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.411818    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-p5pd7"]
Sep 16 19:09:20.428541 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.428521    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"]
Sep 16 19:09:20.435367 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.435345    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6"]
Sep 16 19:09:20.474590 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.474568    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd761dbe-208f-436b-b9d5-47b40ec62b05-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:09:20.474590 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.474591    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:09:20.474732 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.474603    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd761dbe-208f-436b-b9d5-47b40ec62b05-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:09:20.474732 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:20.474614    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4p5j\" (UniqueName: \"kubernetes.io/projected/dd761dbe-208f-436b-b9d5-47b40ec62b05-kube-api-access-n4p5j\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:09:21.268095 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:21.268005    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6" podUID="dd761dbe-208f-436b-b9d5-47b40ec62b05" containerName="main" probeResult="failure" output="Get \"https://10.132.0.25:8001/health\": context deadline exceeded"
Sep 16 19:09:22.160743 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:22.160681    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98d26def-6529-44b4-81f1-2495a5a88631" path="/var/lib/kubelet/pods/98d26def-6529-44b4-81f1-2495a5a88631/volumes"
Sep 16 19:09:22.161094 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:22.161058    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd761dbe-208f-436b-b9d5-47b40ec62b05" path="/var/lib/kubelet/pods/dd761dbe-208f-436b-b9d5-47b40ec62b05/volumes"
Sep 16 19:09:25.271558 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:09:25.271514    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-ckjx6" podUID="dd761dbe-208f-436b-b9d5-47b40ec62b05" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.25:8000/health\": context deadline exceeded"
Sep 16 19:10:36.721427 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.721395    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"]
Sep 16 19:10:36.721990 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.721642    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98d26def-6529-44b4-81f1-2495a5a88631" containerName="main"
Sep 16 19:10:36.721990 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.721654    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="98d26def-6529-44b4-81f1-2495a5a88631" containerName="main"
Sep 16 19:10:36.721990 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.721666    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd761dbe-208f-436b-b9d5-47b40ec62b05" containerName="llm-d-routing-sidecar"
Sep 16 19:10:36.721990 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.721675    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd761dbe-208f-436b-b9d5-47b40ec62b05" containerName="llm-d-routing-sidecar"
Sep 16 19:10:36.721990 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.721712    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd761dbe-208f-436b-b9d5-47b40ec62b05" containerName="main"
Sep 16 19:10:36.721990 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.721720    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd761dbe-208f-436b-b9d5-47b40ec62b05" containerName="main"
Sep 16 19:10:36.721990 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.721775    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="98d26def-6529-44b4-81f1-2495a5a88631" containerName="main"
Sep 16 19:10:36.721990 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.721783    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd761dbe-208f-436b-b9d5-47b40ec62b05" containerName="llm-d-routing-sidecar"
Sep 16 19:10:36.721990 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.721790    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd761dbe-208f-436b-b9d5-47b40ec62b05" containerName="main"
Sep 16 19:10:36.724556 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.724534    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.727946 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.727925    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 19:10:36.728437 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.728417    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-pnvxj\""
Sep 16 19:10:36.729001 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.728978    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 19:10:36.729113 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.729012    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-wd6q2\""
Sep 16 19:10:36.729113 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.729015    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 19:10:36.735549 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.735529    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"]
Sep 16 19:10:36.738379 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.738364    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.741893 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.741871    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"]
Sep 16 19:10:36.760337 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.760317    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"]
Sep 16 19:10:36.781966 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.781942    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.782054 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.781974    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.782054 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.781995    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.782054 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.782019    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.782159 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.782090    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96434b5-6773-46f6-8f7e-f28fde5acfdf-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.782159 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.782118    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.782159 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.782145    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlmht\" (UniqueName: \"kubernetes.io/projected/835a20d5-d174-4de8-ae90-37d43b8ffd7e-kube-api-access-tlmht\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.782252 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.782191    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835a20d5-d174-4de8-ae90-37d43b8ffd7e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.782252 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.782224    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.782252 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.782247    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.782347 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.782270    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jskf4\" (UniqueName: \"kubernetes.io/projected/b96434b5-6773-46f6-8f7e-f28fde5acfdf-kube-api-access-jskf4\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.782347 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.782292    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.882769 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.882747    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.882866 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.882779    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.882866 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.882803    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jskf4\" (UniqueName: \"kubernetes.io/projected/b96434b5-6773-46f6-8f7e-f28fde5acfdf-kube-api-access-jskf4\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.882866 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.882851    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.882972 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.882893    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.882972 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.882922    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.882972 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.882942    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.882972 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.882959    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.883151 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.882987    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96434b5-6773-46f6-8f7e-f28fde5acfdf-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.883151 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.883018    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.883151 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.883060    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlmht\" (UniqueName: \"kubernetes.io/projected/835a20d5-d174-4de8-ae90-37d43b8ffd7e-kube-api-access-tlmht\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.883151 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.883086    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835a20d5-d174-4de8-ae90-37d43b8ffd7e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.883358 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.883223    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.883358 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.883279    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.883358 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.883325    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.883358 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.883341    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.883558 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.883506    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.883751 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.883724    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.885470 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.885441    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835a20d5-d174-4de8-ae90-37d43b8ffd7e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.885555 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.885531    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.885609 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.885531    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96434b5-6773-46f6-8f7e-f28fde5acfdf-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:36.885609 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.885583    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.893280 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.893250    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlmht\" (UniqueName: \"kubernetes.io/projected/835a20d5-d174-4de8-ae90-37d43b8ffd7e-kube-api-access-tlmht\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:36.894590 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:36.894570    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jskf4\" (UniqueName: \"kubernetes.io/projected/b96434b5-6773-46f6-8f7e-f28fde5acfdf-kube-api-access-jskf4\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:37.033921 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:37.033895    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:37.047709 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:37.047661    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:37.324213 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:37.324189    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"]
Sep 16 19:10:37.326028 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:10:37.325995    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835a20d5_d174_4de8_ae90_37d43b8ffd7e.slice/crio-778a420b7a7d78e7bb57a23448acc369dceeeef14da29fa38609e63c22db3e05 WatchSource:0}: Error finding container 778a420b7a7d78e7bb57a23448acc369dceeeef14da29fa38609e63c22db3e05: Status 404 returned error can't find the container with id 778a420b7a7d78e7bb57a23448acc369dceeeef14da29fa38609e63c22db3e05
Sep 16 19:10:37.361381 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:37.361310    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"]
Sep 16 19:10:37.363046 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:10:37.363024    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96434b5_6773_46f6_8f7e_f28fde5acfdf.slice/crio-f8825aca903f0183443d87395f9d280a6abe340dc32d88b35c6a15f53323efbf WatchSource:0}: Error finding container f8825aca903f0183443d87395f9d280a6abe340dc32d88b35c6a15f53323efbf: Status 404 returned error can't find the container with id f8825aca903f0183443d87395f9d280a6abe340dc32d88b35c6a15f53323efbf
Sep 16 19:10:37.646111 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:37.646026    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c" event={"ID":"835a20d5-d174-4de8-ae90-37d43b8ffd7e","Type":"ContainerStarted","Data":"8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4"}
Sep 16 19:10:37.646272 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:37.646177    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:37.646272 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:37.646219    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c" event={"ID":"835a20d5-d174-4de8-ae90-37d43b8ffd7e","Type":"ContainerStarted","Data":"778a420b7a7d78e7bb57a23448acc369dceeeef14da29fa38609e63c22db3e05"}
Sep 16 19:10:37.647608 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:37.647581    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x" event={"ID":"b96434b5-6773-46f6-8f7e-f28fde5acfdf","Type":"ContainerStarted","Data":"ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657"}
Sep 16 19:10:37.647727 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:37.647629    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x" event={"ID":"b96434b5-6773-46f6-8f7e-f28fde5acfdf","Type":"ContainerStarted","Data":"f8825aca903f0183443d87395f9d280a6abe340dc32d88b35c6a15f53323efbf"}
Sep 16 19:10:37.691569 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:37.691530    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x" podStartSLOduration=1.69151719 podStartE2EDuration="1.69151719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:37.691395326 +0000 UTC m=+1564.075954550" watchObservedRunningTime="2026-09-16 19:10:37.69151719 +0000 UTC m=+1564.076076412"
Sep 16 19:10:38.047999 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:38.047972    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:38.653111 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:38.653075    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c" event={"ID":"835a20d5-d174-4de8-ae90-37d43b8ffd7e","Type":"ContainerStarted","Data":"12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a"}
Sep 16 19:10:38.675414 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:38.675360    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c" podStartSLOduration=2.675341955 podStartE2EDuration="2.675341955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:38.673508352 +0000 UTC m=+1565.058067577" watchObservedRunningTime="2026-09-16 19:10:38.675341955 +0000 UTC m=+1565.059901180"
Sep 16 19:10:39.034171 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:39.034148    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:47.034902 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:47.034870    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:47.047582 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:47.047551    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:47.047795 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:47.047780    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:47.053075 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:47.053050    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:47.060120 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:47.060099    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:10:47.696291 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:47.696264    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:10:47.697036 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:10:47.697013    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:11:11.845737 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:11.845680    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"]
Sep 16 19:11:11.848871 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:11.848844    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"]
Sep 16 19:11:11.854541 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:11:11.854508    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:11:11.854541 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:11.854541 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:11.854541 ip-10-0-135-244 kubenswrapper[2570]:  > containerID="ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657" cmd=["/bin/sleep","15"]
Sep 16 19:11:11.854767 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:11.854554    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x" message="" err=<
Sep 16 19:11:11.854767 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:11.854767 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:11.854767 ip-10-0-135-244 kubenswrapper[2570]:  >
Sep 16 19:11:11.854767 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:11:11.854567    2570 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:11:11.854767 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:11.854767 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:11.854767 ip-10-0-135-244 kubenswrapper[2570]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x" podUID="b96434b5-6773-46f6-8f7e-f28fde5acfdf" containerName="main" containerID="cri-o://ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657"
Sep 16 19:11:11.854767 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:11.854598    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x" podUID="b96434b5-6773-46f6-8f7e-f28fde5acfdf" containerName="main" containerID="cri-o://ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657" gracePeriod=60
Sep 16 19:11:11.856635 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:11:11.856604    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:11:11.856635 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:11.856635 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:11.856635 ip-10-0-135-244 kubenswrapper[2570]:  > containerID="12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a" cmd=["/bin/sleep","15"]
Sep 16 19:11:11.856891 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:11.856655    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c" message="" err=<
Sep 16 19:11:11.856891 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:11.856891 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:11.856891 ip-10-0-135-244 kubenswrapper[2570]:  >
Sep 16 19:11:11.856891 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:11:11.856713    2570 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:11:11.856891 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:11.856891 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:11.856891 ip-10-0-135-244 kubenswrapper[2570]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c" podUID="835a20d5-d174-4de8-ae90-37d43b8ffd7e" containerName="main" containerID="cri-o://12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a"
Sep 16 19:11:11.856891 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:11.856757    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c" podUID="835a20d5-d174-4de8-ae90-37d43b8ffd7e" containerName="main" containerID="cri-o://12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a" gracePeriod=60
Sep 16 19:11:12.072626 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.072576    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c" podUID="835a20d5-d174-4de8-ae90-37d43b8ffd7e" containerName="llm-d-routing-sidecar" containerID="cri-o://8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4" gracePeriod=60
Sep 16 19:11:12.206931 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.206903    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:11:12.327204 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.327178    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:11:12.352492 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.352471    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-dshm\") pod \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") "
Sep 16 19:11:12.352590 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.352501    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-tmp-dir\") pod \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") "
Sep 16 19:11:12.352590 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.352535    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-model-cache\") pod \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") "
Sep 16 19:11:12.352590 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.352575    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-home\") pod \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") "
Sep 16 19:11:12.352764 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.352591    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jskf4\" (UniqueName: \"kubernetes.io/projected/b96434b5-6773-46f6-8f7e-f28fde5acfdf-kube-api-access-jskf4\") pod \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") "
Sep 16 19:11:12.352764 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.352611    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96434b5-6773-46f6-8f7e-f28fde5acfdf-tls-certs\") pod \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\" (UID: \"b96434b5-6773-46f6-8f7e-f28fde5acfdf\") "
Sep 16 19:11:12.352913 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.352862    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-model-cache" (OuterVolumeSpecName: "model-cache") pod "b96434b5-6773-46f6-8f7e-f28fde5acfdf" (UID: "b96434b5-6773-46f6-8f7e-f28fde5acfdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:12.353172 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.352916    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-home" (OuterVolumeSpecName: "home") pod "b96434b5-6773-46f6-8f7e-f28fde5acfdf" (UID: "b96434b5-6773-46f6-8f7e-f28fde5acfdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:12.353323 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.353297    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b96434b5-6773-46f6-8f7e-f28fde5acfdf" (UID: "b96434b5-6773-46f6-8f7e-f28fde5acfdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:12.355315 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.355290    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-dshm" (OuterVolumeSpecName: "dshm") pod "b96434b5-6773-46f6-8f7e-f28fde5acfdf" (UID: "b96434b5-6773-46f6-8f7e-f28fde5acfdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:12.355417 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.355372    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96434b5-6773-46f6-8f7e-f28fde5acfdf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b96434b5-6773-46f6-8f7e-f28fde5acfdf" (UID: "b96434b5-6773-46f6-8f7e-f28fde5acfdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:12.355514 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.355495    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96434b5-6773-46f6-8f7e-f28fde5acfdf-kube-api-access-jskf4" (OuterVolumeSpecName: "kube-api-access-jskf4") pod "b96434b5-6773-46f6-8f7e-f28fde5acfdf" (UID: "b96434b5-6773-46f6-8f7e-f28fde5acfdf"). InnerVolumeSpecName "kube-api-access-jskf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:12.453807 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.453751    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835a20d5-d174-4de8-ae90-37d43b8ffd7e-tls-certs\") pod \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") "
Sep 16 19:11:12.453807 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.453781    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-home\") pod \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") "
Sep 16 19:11:12.453964 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.453811    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-model-cache\") pod \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") "
Sep 16 19:11:12.453964 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.453840    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlmht\" (UniqueName: \"kubernetes.io/projected/835a20d5-d174-4de8-ae90-37d43b8ffd7e-kube-api-access-tlmht\") pod \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") "
Sep 16 19:11:12.453964 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.453866    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-tmp-dir\") pod \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") "
Sep 16 19:11:12.453964 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.453885    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-dshm\") pod \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\" (UID: \"835a20d5-d174-4de8-ae90-37d43b8ffd7e\") "
Sep 16 19:11:12.454168 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.454033    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:11:12.454168 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.454061    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-home" (OuterVolumeSpecName: "home") pod "835a20d5-d174-4de8-ae90-37d43b8ffd7e" (UID: "835a20d5-d174-4de8-ae90-37d43b8ffd7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:12.454168 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.454082    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:11:12.454168 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.454119    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:11:12.454168 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.454132    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96434b5-6773-46f6-8f7e-f28fde5acfdf-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:11:12.454168 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.454146    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jskf4\" (UniqueName: \"kubernetes.io/projected/b96434b5-6773-46f6-8f7e-f28fde5acfdf-kube-api-access-jskf4\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:11:12.454168 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.454159    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96434b5-6773-46f6-8f7e-f28fde5acfdf-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:11:12.454168 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.454075    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "835a20d5-d174-4de8-ae90-37d43b8ffd7e" (UID: "835a20d5-d174-4de8-ae90-37d43b8ffd7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:12.454550 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.454331    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "835a20d5-d174-4de8-ae90-37d43b8ffd7e" (UID: "835a20d5-d174-4de8-ae90-37d43b8ffd7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:12.455806 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.455783    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835a20d5-d174-4de8-ae90-37d43b8ffd7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "835a20d5-d174-4de8-ae90-37d43b8ffd7e" (UID: "835a20d5-d174-4de8-ae90-37d43b8ffd7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:12.455885 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.455869    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-dshm" (OuterVolumeSpecName: "dshm") pod "835a20d5-d174-4de8-ae90-37d43b8ffd7e" (UID: "835a20d5-d174-4de8-ae90-37d43b8ffd7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:12.455925 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.455901    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835a20d5-d174-4de8-ae90-37d43b8ffd7e-kube-api-access-tlmht" (OuterVolumeSpecName: "kube-api-access-tlmht") pod "835a20d5-d174-4de8-ae90-37d43b8ffd7e" (UID: "835a20d5-d174-4de8-ae90-37d43b8ffd7e"). InnerVolumeSpecName "kube-api-access-tlmht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:12.554537 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.554515    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:11:12.554537 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.554536    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:11:12.554645 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.554544    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835a20d5-d174-4de8-ae90-37d43b8ffd7e-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:11:12.554645 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.554551    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:11:12.554645 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.554559    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835a20d5-d174-4de8-ae90-37d43b8ffd7e-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:11:12.554645 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.554567    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlmht\" (UniqueName: \"kubernetes.io/projected/835a20d5-d174-4de8-ae90-37d43b8ffd7e-kube-api-access-tlmht\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:11:12.787912 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.787887    2570 generic.go:356] "Generic (PLEG): container finished" podID="835a20d5-d174-4de8-ae90-37d43b8ffd7e" containerID="12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a" exitCode=0
Sep 16 19:11:12.787912 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.787908    2570 generic.go:356] "Generic (PLEG): container finished" podID="835a20d5-d174-4de8-ae90-37d43b8ffd7e" containerID="8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4" exitCode=0
Sep 16 19:11:12.788082 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.787932    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"
Sep 16 19:11:12.788082 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.787973    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c" event={"ID":"835a20d5-d174-4de8-ae90-37d43b8ffd7e","Type":"ContainerDied","Data":"12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a"}
Sep 16 19:11:12.788082 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.788029    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c" event={"ID":"835a20d5-d174-4de8-ae90-37d43b8ffd7e","Type":"ContainerDied","Data":"8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4"}
Sep 16 19:11:12.788082 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.788049    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c" event={"ID":"835a20d5-d174-4de8-ae90-37d43b8ffd7e","Type":"ContainerDied","Data":"778a420b7a7d78e7bb57a23448acc369dceeeef14da29fa38609e63c22db3e05"}
Sep 16 19:11:12.788082 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.788079    2570 scope.go:117] "RemoveContainer" containerID="12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a"
Sep 16 19:11:12.789415 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.789394    2570 generic.go:356] "Generic (PLEG): container finished" podID="b96434b5-6773-46f6-8f7e-f28fde5acfdf" containerID="ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657" exitCode=0
Sep 16 19:11:12.789498 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.789430    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x" event={"ID":"b96434b5-6773-46f6-8f7e-f28fde5acfdf","Type":"ContainerDied","Data":"ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657"}
Sep 16 19:11:12.789498 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.789484    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x" event={"ID":"b96434b5-6773-46f6-8f7e-f28fde5acfdf","Type":"ContainerDied","Data":"f8825aca903f0183443d87395f9d280a6abe340dc32d88b35c6a15f53323efbf"}
Sep 16 19:11:12.789578 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.789442    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"
Sep 16 19:11:12.797203 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.797181    2570 scope.go:117] "RemoveContainer" containerID="8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4"
Sep 16 19:11:12.804511 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.804492    2570 scope.go:117] "RemoveContainer" containerID="12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a"
Sep 16 19:11:12.804778 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:11:12.804752    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a\": container with ID starting with 12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a not found: ID does not exist" containerID="12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a"
Sep 16 19:11:12.804841 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.804778    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a"} err="failed to get container status \"12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a\": rpc error: code = NotFound desc = could not find container \"12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a\": container with ID starting with 12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a not found: ID does not exist"
Sep 16 19:11:12.804841 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.804795    2570 scope.go:117] "RemoveContainer" containerID="8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4"
Sep 16 19:11:12.805060 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:11:12.805037    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4\": container with ID starting with 8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4 not found: ID does not exist" containerID="8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4"
Sep 16 19:11:12.805138 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.805064    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4"} err="failed to get container status \"8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4\": rpc error: code = NotFound desc = could not find container \"8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4\": container with ID starting with 8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4 not found: ID does not exist"
Sep 16 19:11:12.805138 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.805080    2570 scope.go:117] "RemoveContainer" containerID="12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a"
Sep 16 19:11:12.805322 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.805302    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a"} err="failed to get container status \"12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a\": rpc error: code = NotFound desc = could not find container \"12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a\": container with ID starting with 12f99f445944000bb00d826bfec052ac03635ffd050306712bb4c6b61d546b9a not found: ID does not exist"
Sep 16 19:11:12.805364 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.805324    2570 scope.go:117] "RemoveContainer" containerID="8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4"
Sep 16 19:11:12.805546 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.805528    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4"} err="failed to get container status \"8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4\": rpc error: code = NotFound desc = could not find container \"8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4\": container with ID starting with 8be090bffd13ad7b600d3fdef88480bf1d381aecffd1979efcd052cddb4fb9f4 not found: ID does not exist"
Sep 16 19:11:12.805592 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.805548    2570 scope.go:117] "RemoveContainer" containerID="ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657"
Sep 16 19:11:12.812511 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.812492    2570 scope.go:117] "RemoveContainer" containerID="ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657"
Sep 16 19:11:12.812853 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:11:12.812834    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657\": container with ID starting with ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657 not found: ID does not exist" containerID="ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657"
Sep 16 19:11:12.812924 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.812857    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657"} err="failed to get container status \"ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657\": rpc error: code = NotFound desc = could not find container \"ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657\": container with ID starting with ca28fed7ebea840bb967cfb609b918139b8d9685a5ef21c3ec06fb76961c2657 not found: ID does not exist"
Sep 16 19:11:12.813800 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.813783    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"]
Sep 16 19:11:12.821023 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.821005    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qjr4c"]
Sep 16 19:11:12.838198 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.838173    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"]
Sep 16 19:11:12.839189 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:12.839172    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64445zg9x"]
Sep 16 19:11:14.160207 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:14.160177    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="835a20d5-d174-4de8-ae90-37d43b8ffd7e" path="/var/lib/kubelet/pods/835a20d5-d174-4de8-ae90-37d43b8ffd7e/volumes"
Sep 16 19:11:14.160566 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:11:14.160552    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b96434b5-6773-46f6-8f7e-f28fde5acfdf" path="/var/lib/kubelet/pods/b96434b5-6773-46f6-8f7e-f28fde5acfdf/volumes"
Sep 16 19:12:28.170213 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.170180    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"]
Sep 16 19:12:28.172493 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.170415    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835a20d5-d174-4de8-ae90-37d43b8ffd7e" containerName="llm-d-routing-sidecar"
Sep 16 19:12:28.172493 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.170425    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="835a20d5-d174-4de8-ae90-37d43b8ffd7e" containerName="llm-d-routing-sidecar"
Sep 16 19:12:28.172493 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.170434    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835a20d5-d174-4de8-ae90-37d43b8ffd7e" containerName="main"
Sep 16 19:12:28.172493 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.170440    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="835a20d5-d174-4de8-ae90-37d43b8ffd7e" containerName="main"
Sep 16 19:12:28.172493 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.170449    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96434b5-6773-46f6-8f7e-f28fde5acfdf" containerName="main"
Sep 16 19:12:28.172493 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.170455    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96434b5-6773-46f6-8f7e-f28fde5acfdf" containerName="main"
Sep 16 19:12:28.172493 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.170500    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b96434b5-6773-46f6-8f7e-f28fde5acfdf" containerName="main"
Sep 16 19:12:28.172493 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.170505    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="835a20d5-d174-4de8-ae90-37d43b8ffd7e" containerName="main"
Sep 16 19:12:28.172493 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.170512    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="835a20d5-d174-4de8-ae90-37d43b8ffd7e" containerName="llm-d-routing-sidecar"
Sep 16 19:12:28.173351 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.173333    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.179665 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.179642    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-c2vgg\""
Sep 16 19:12:28.179792 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.179640    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:12:28.180095 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.180070    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 19:12:28.180225 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.180009    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 19:12:28.180319 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.180302    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 19:12:28.180427 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.180405    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 19:12:28.194710 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.194677    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"]
Sep 16 19:12:28.269615 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.269591    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-home\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.269765 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.269620    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slxzs\" (UniqueName: \"kubernetes.io/projected/9b520559-0fc1-45ae-807c-550a7ebb92fa-kube-api-access-slxzs\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.269765 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.269643    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.269765 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.269661    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b520559-0fc1-45ae-807c-550a7ebb92fa-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.269882 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.269764    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-model-cache\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.269882 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.269797    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-dshm\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.269882 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.269812    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.269882 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.269826    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b520559-0fc1-45ae-807c-550a7ebb92fa-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.370396 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.370363    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-home\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.370557 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.370405    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slxzs\" (UniqueName: \"kubernetes.io/projected/9b520559-0fc1-45ae-807c-550a7ebb92fa-kube-api-access-slxzs\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.370557 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.370428    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.370557 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.370446    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b520559-0fc1-45ae-807c-550a7ebb92fa-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.370557 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.370481    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-model-cache\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.370557 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.370504    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-dshm\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.370557 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.370521    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.370557 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.370539    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b520559-0fc1-45ae-807c-550a7ebb92fa-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.370917 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.370841    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.370917 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.370852    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-home\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.371023 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.370942    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-model-cache\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.371023 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.370980    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.371297 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.371272    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b520559-0fc1-45ae-807c-550a7ebb92fa-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.372949 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.372920    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-dshm\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.373031 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.372988    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b520559-0fc1-45ae-807c-550a7ebb92fa-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.380386 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.380354    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slxzs\" (UniqueName: \"kubernetes.io/projected/9b520559-0fc1-45ae-807c-550a7ebb92fa-kube-api-access-slxzs\") pod \"tokenizer-clean-path-test-kserve-7cd95947f-m9f4p\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.485044 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.484976    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:28.605193 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.605159    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"]
Sep 16 19:12:28.608486 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.608471    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"
Sep 16 19:12:28.621165 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.621144    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-x9wh6\""
Sep 16 19:12:28.635420 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.635393    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"]
Sep 16 19:12:28.672313 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.672292    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c513fc5-dffc-4ddc-9688-41e9f46307a2-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln\" (UID: \"0c513fc5-dffc-4ddc-9688-41e9f46307a2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"
Sep 16 19:12:28.672412 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.672334    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrtc9\" (UniqueName: \"kubernetes.io/projected/0c513fc5-dffc-4ddc-9688-41e9f46307a2-kube-api-access-xrtc9\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln\" (UID: \"0c513fc5-dffc-4ddc-9688-41e9f46307a2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"
Sep 16 19:12:28.768188 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.768121    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"]
Sep 16 19:12:28.771278 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:12:28.771253    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b520559_0fc1_45ae_807c_550a7ebb92fa.slice/crio-073c23d741950b49b0d420265ef2f6c1fe9bf17a2f553fa8d11f3ec3bd714fda WatchSource:0}: Error finding container 073c23d741950b49b0d420265ef2f6c1fe9bf17a2f553fa8d11f3ec3bd714fda: Status 404 returned error can't find the container with id 073c23d741950b49b0d420265ef2f6c1fe9bf17a2f553fa8d11f3ec3bd714fda
Sep 16 19:12:28.772886 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.772804    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrtc9\" (UniqueName: \"kubernetes.io/projected/0c513fc5-dffc-4ddc-9688-41e9f46307a2-kube-api-access-xrtc9\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln\" (UID: \"0c513fc5-dffc-4ddc-9688-41e9f46307a2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"
Sep 16 19:12:28.772954 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.772885    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c513fc5-dffc-4ddc-9688-41e9f46307a2-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln\" (UID: \"0c513fc5-dffc-4ddc-9688-41e9f46307a2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"
Sep 16 19:12:28.775241 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.775217    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c513fc5-dffc-4ddc-9688-41e9f46307a2-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln\" (UID: \"0c513fc5-dffc-4ddc-9688-41e9f46307a2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"
Sep 16 19:12:28.782943 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.782922    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrtc9\" (UniqueName: \"kubernetes.io/projected/0c513fc5-dffc-4ddc-9688-41e9f46307a2-kube-api-access-xrtc9\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln\" (UID: \"0c513fc5-dffc-4ddc-9688-41e9f46307a2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"
Sep 16 19:12:28.918003 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:28.917980    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"
Sep 16 19:12:29.057348 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:29.057320    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p" event={"ID":"9b520559-0fc1-45ae-807c-550a7ebb92fa","Type":"ContainerStarted","Data":"2d3eac5288eae0615e09bede6e5581732b6b7a9855de5f6323d4f2972d6a725d"}
Sep 16 19:12:29.057496 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:29.057366    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p" event={"ID":"9b520559-0fc1-45ae-807c-550a7ebb92fa","Type":"ContainerStarted","Data":"073c23d741950b49b0d420265ef2f6c1fe9bf17a2f553fa8d11f3ec3bd714fda"}
Sep 16 19:12:29.189726 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:29.189703    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"]
Sep 16 19:12:29.191255 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:12:29.191226    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c513fc5_dffc_4ddc_9688_41e9f46307a2.slice/crio-9aab350a09018867065f618c0f8c2005879a0a1d6453fd9680b3893ae91202ec WatchSource:0}: Error finding container 9aab350a09018867065f618c0f8c2005879a0a1d6453fd9680b3893ae91202ec: Status 404 returned error can't find the container with id 9aab350a09018867065f618c0f8c2005879a0a1d6453fd9680b3893ae91202ec
Sep 16 19:12:30.064752 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:30.064652    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln" event={"ID":"0c513fc5-dffc-4ddc-9688-41e9f46307a2","Type":"ContainerStarted","Data":"0a59ca3ffbfd77dc5ea2ba81538a0342072d6fdee093f53766b024537e3a07c4"}
Sep 16 19:12:30.064752 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:30.064731    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"
Sep 16 19:12:30.064948 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:30.064755    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln" event={"ID":"0c513fc5-dffc-4ddc-9688-41e9f46307a2","Type":"ContainerStarted","Data":"9aab350a09018867065f618c0f8c2005879a0a1d6453fd9680b3893ae91202ec"}
Sep 16 19:12:30.086954 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:30.086906    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln" podStartSLOduration=2.08688996 podStartE2EDuration="2.08688996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:30.086193659 +0000 UTC m=+1676.470752883" watchObservedRunningTime="2026-09-16 19:12:30.08688996 +0000 UTC m=+1676.471449183"
Sep 16 19:12:31.070113 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:31.070083    2570 generic.go:356] "Generic (PLEG): container finished" podID="9b520559-0fc1-45ae-807c-550a7ebb92fa" containerID="2d3eac5288eae0615e09bede6e5581732b6b7a9855de5f6323d4f2972d6a725d" exitCode=0
Sep 16 19:12:31.070444 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:31.070165    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p" event={"ID":"9b520559-0fc1-45ae-807c-550a7ebb92fa","Type":"ContainerDied","Data":"2d3eac5288eae0615e09bede6e5581732b6b7a9855de5f6323d4f2972d6a725d"}
Sep 16 19:12:32.075085 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:32.075051    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p" event={"ID":"9b520559-0fc1-45ae-807c-550a7ebb92fa","Type":"ContainerStarted","Data":"b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7"}
Sep 16 19:12:32.485630 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:32.485539    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:38.486222 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:38.486188    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:38.498904 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:38.498875    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:12:38.524122 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:38.524071    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p" podStartSLOduration=8.399512719 podStartE2EDuration="10.524056917s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.124544198s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:32.109642407 +0000 UTC m=+1678.494201630" watchObservedRunningTime="2026-09-16 19:12:38.524056917 +0000 UTC m=+1684.908616140"
Sep 16 19:12:39.111818 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:12:39.111792    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:13:01.072138 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:13:01.072061    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"
Sep 16 19:14:45.842219 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:45.842181    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"]
Sep 16 19:14:45.844045 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:45.844005    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"]
Sep 16 19:14:45.851587 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:14:45.851556    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:14:45.851587 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:14:45.851587 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:14:45.851587 ip-10-0-135-244 kubenswrapper[2570]:  > containerID="b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7" cmd=["/bin/sleep","15"]
Sep 16 19:14:45.851931 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:45.851610    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p" message="" err=<
Sep 16 19:14:45.851931 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:14:45.851931 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:14:45.851931 ip-10-0-135-244 kubenswrapper[2570]:  >
Sep 16 19:14:45.851931 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:14:45.851627    2570 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:14:45.851931 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:14:45.851931 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:14:45.851931 ip-10-0-135-244 kubenswrapper[2570]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p" podUID="9b520559-0fc1-45ae-807c-550a7ebb92fa" containerName="main" containerID="cri-o://b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7"
Sep 16 19:14:45.851931 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:45.851668    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p" podUID="9b520559-0fc1-45ae-807c-550a7ebb92fa" containerName="main" containerID="cri-o://b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7" gracePeriod=60
Sep 16 19:14:46.101033 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.100951    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p" podUID="9b520559-0fc1-45ae-807c-550a7ebb92fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.29:8000/health\": dial tcp 10.132.0.29:8000: connect: connection refused"
Sep 16 19:14:46.198306 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.198287    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:14:46.258011 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.257983    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-tmp-dir\") pod \"9b520559-0fc1-45ae-807c-550a7ebb92fa\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") "
Sep 16 19:14:46.258142 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.258030    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-kserve-provision-location\") pod \"9b520559-0fc1-45ae-807c-550a7ebb92fa\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") "
Sep 16 19:14:46.258142 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.258062    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slxzs\" (UniqueName: \"kubernetes.io/projected/9b520559-0fc1-45ae-807c-550a7ebb92fa-kube-api-access-slxzs\") pod \"9b520559-0fc1-45ae-807c-550a7ebb92fa\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") "
Sep 16 19:14:46.258142 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.258085    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-dshm\") pod \"9b520559-0fc1-45ae-807c-550a7ebb92fa\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") "
Sep 16 19:14:46.258142 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.258113    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-model-cache\") pod \"9b520559-0fc1-45ae-807c-550a7ebb92fa\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") "
Sep 16 19:14:46.258142 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.258136    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b520559-0fc1-45ae-807c-550a7ebb92fa-tls-certs\") pod \"9b520559-0fc1-45ae-807c-550a7ebb92fa\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") "
Sep 16 19:14:46.258391 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.258182    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-home\") pod \"9b520559-0fc1-45ae-807c-550a7ebb92fa\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") "
Sep 16 19:14:46.258391 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.258238    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b520559-0fc1-45ae-807c-550a7ebb92fa-cabundle-cert\") pod \"9b520559-0fc1-45ae-807c-550a7ebb92fa\" (UID: \"9b520559-0fc1-45ae-807c-550a7ebb92fa\") "
Sep 16 19:14:46.258391 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.258276    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b520559-0fc1-45ae-807c-550a7ebb92fa" (UID: "9b520559-0fc1-45ae-807c-550a7ebb92fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:46.258391 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.258353    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b520559-0fc1-45ae-807c-550a7ebb92fa" (UID: "9b520559-0fc1-45ae-807c-550a7ebb92fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:46.258578 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.258505    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-home" (OuterVolumeSpecName: "home") pod "9b520559-0fc1-45ae-807c-550a7ebb92fa" (UID: "9b520559-0fc1-45ae-807c-550a7ebb92fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:46.258578 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.258551    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.258578 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.258570    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.258753 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.258726    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b520559-0fc1-45ae-807c-550a7ebb92fa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b520559-0fc1-45ae-807c-550a7ebb92fa" (UID: "9b520559-0fc1-45ae-807c-550a7ebb92fa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:46.260624 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.260598    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-dshm" (OuterVolumeSpecName: "dshm") pod "9b520559-0fc1-45ae-807c-550a7ebb92fa" (UID: "9b520559-0fc1-45ae-807c-550a7ebb92fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:46.260883 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.260725    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b520559-0fc1-45ae-807c-550a7ebb92fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b520559-0fc1-45ae-807c-550a7ebb92fa" (UID: "9b520559-0fc1-45ae-807c-550a7ebb92fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:46.260883 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.260755    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b520559-0fc1-45ae-807c-550a7ebb92fa-kube-api-access-slxzs" (OuterVolumeSpecName: "kube-api-access-slxzs") pod "9b520559-0fc1-45ae-807c-550a7ebb92fa" (UID: "9b520559-0fc1-45ae-807c-550a7ebb92fa"). InnerVolumeSpecName "kube-api-access-slxzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:46.278531 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.278508    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b520559-0fc1-45ae-807c-550a7ebb92fa" (UID: "9b520559-0fc1-45ae-807c-550a7ebb92fa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:46.359842 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.359777    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-kserve-provision-location\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.359842 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.359801    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slxzs\" (UniqueName: \"kubernetes.io/projected/9b520559-0fc1-45ae-807c-550a7ebb92fa-kube-api-access-slxzs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.359842 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.359810    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.359842 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.359819    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b520559-0fc1-45ae-807c-550a7ebb92fa-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.359842 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.359828    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b520559-0fc1-45ae-807c-550a7ebb92fa-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.359842 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.359838    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b520559-0fc1-45ae-807c-550a7ebb92fa-cabundle-cert\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:14:46.554895 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.554865    2570 generic.go:356] "Generic (PLEG): container finished" podID="9b520559-0fc1-45ae-807c-550a7ebb92fa" containerID="b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7" exitCode=0
Sep 16 19:14:46.555011 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.554913    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"
Sep 16 19:14:46.555011 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.554947    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p" event={"ID":"9b520559-0fc1-45ae-807c-550a7ebb92fa","Type":"ContainerDied","Data":"b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7"}
Sep 16 19:14:46.555011 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.554999    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p" event={"ID":"9b520559-0fc1-45ae-807c-550a7ebb92fa","Type":"ContainerDied","Data":"073c23d741950b49b0d420265ef2f6c1fe9bf17a2f553fa8d11f3ec3bd714fda"}
Sep 16 19:14:46.555154 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.555022    2570 scope.go:117] "RemoveContainer" containerID="b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7"
Sep 16 19:14:46.563988 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.563972    2570 scope.go:117] "RemoveContainer" containerID="2d3eac5288eae0615e09bede6e5581732b6b7a9855de5f6323d4f2972d6a725d"
Sep 16 19:14:46.571004 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.570987    2570 scope.go:117] "RemoveContainer" containerID="b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7"
Sep 16 19:14:46.571255 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:14:46.571240    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7\": container with ID starting with b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7 not found: ID does not exist" containerID="b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7"
Sep 16 19:14:46.571297 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.571264    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7"} err="failed to get container status \"b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7\": rpc error: code = NotFound desc = could not find container \"b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7\": container with ID starting with b6b7d839393318e21ce488d05b4ecb78229dee6e0e08efa36bdf670f878f19f7 not found: ID does not exist"
Sep 16 19:14:46.571297 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.571279    2570 scope.go:117] "RemoveContainer" containerID="2d3eac5288eae0615e09bede6e5581732b6b7a9855de5f6323d4f2972d6a725d"
Sep 16 19:14:46.571496 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:14:46.571480    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d3eac5288eae0615e09bede6e5581732b6b7a9855de5f6323d4f2972d6a725d\": container with ID starting with 2d3eac5288eae0615e09bede6e5581732b6b7a9855de5f6323d4f2972d6a725d not found: ID does not exist" containerID="2d3eac5288eae0615e09bede6e5581732b6b7a9855de5f6323d4f2972d6a725d"
Sep 16 19:14:46.571540 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.571502    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d3eac5288eae0615e09bede6e5581732b6b7a9855de5f6323d4f2972d6a725d"} err="failed to get container status \"2d3eac5288eae0615e09bede6e5581732b6b7a9855de5f6323d4f2972d6a725d\": rpc error: code = NotFound desc = could not find container \"2d3eac5288eae0615e09bede6e5581732b6b7a9855de5f6323d4f2972d6a725d\": container with ID starting with 2d3eac5288eae0615e09bede6e5581732b6b7a9855de5f6323d4f2972d6a725d not found: ID does not exist"
Sep 16 19:14:46.576831 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.576809    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"]
Sep 16 19:14:46.586196 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:46.583979    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7cd95947f-m9f4p"]
Sep 16 19:14:48.160511 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:48.160473    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b520559-0fc1-45ae-807c-550a7ebb92fa" path="/var/lib/kubelet/pods/9b520559-0fc1-45ae-807c-550a7ebb92fa/volumes"
Sep 16 19:14:51.071654 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:14:51.071617    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln" podUID="0c513fc5-dffc-4ddc-9688-41e9f46307a2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:15:00.863101 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:00.863049    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln" podUID="0c513fc5-dffc-4ddc-9688-41e9f46307a2" containerName="main" containerID="cri-o://0a59ca3ffbfd77dc5ea2ba81538a0342072d6fdee093f53766b024537e3a07c4" gracePeriod=45
Sep 16 19:15:01.072698 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:01.072640    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln" podUID="0c513fc5-dffc-4ddc-9688-41e9f46307a2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:15:11.071912 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:11.071872    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln" podUID="0c513fc5-dffc-4ddc-9688-41e9f46307a2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:15:11.072264 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:11.071987    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"
Sep 16 19:15:21.072420 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:21.072385    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln" podUID="0c513fc5-dffc-4ddc-9688-41e9f46307a2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:15:31.111099 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.111075    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"
Sep 16 19:15:31.170728 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.170657    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c513fc5-dffc-4ddc-9688-41e9f46307a2-tls-certs\") pod \"0c513fc5-dffc-4ddc-9688-41e9f46307a2\" (UID: \"0c513fc5-dffc-4ddc-9688-41e9f46307a2\") "
Sep 16 19:15:31.170864 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.170743    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrtc9\" (UniqueName: \"kubernetes.io/projected/0c513fc5-dffc-4ddc-9688-41e9f46307a2-kube-api-access-xrtc9\") pod \"0c513fc5-dffc-4ddc-9688-41e9f46307a2\" (UID: \"0c513fc5-dffc-4ddc-9688-41e9f46307a2\") "
Sep 16 19:15:31.172622 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.172591    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c513fc5-dffc-4ddc-9688-41e9f46307a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c513fc5-dffc-4ddc-9688-41e9f46307a2" (UID: "0c513fc5-dffc-4ddc-9688-41e9f46307a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:31.172731 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.172599    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c513fc5-dffc-4ddc-9688-41e9f46307a2-kube-api-access-xrtc9" (OuterVolumeSpecName: "kube-api-access-xrtc9") pod "0c513fc5-dffc-4ddc-9688-41e9f46307a2" (UID: "0c513fc5-dffc-4ddc-9688-41e9f46307a2"). InnerVolumeSpecName "kube-api-access-xrtc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:31.272162 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.272125    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c513fc5-dffc-4ddc-9688-41e9f46307a2-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:15:31.272162 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.272162    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrtc9\" (UniqueName: \"kubernetes.io/projected/0c513fc5-dffc-4ddc-9688-41e9f46307a2-kube-api-access-xrtc9\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:15:31.715535 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.715498    2570 generic.go:356] "Generic (PLEG): container finished" podID="0c513fc5-dffc-4ddc-9688-41e9f46307a2" containerID="0a59ca3ffbfd77dc5ea2ba81538a0342072d6fdee093f53766b024537e3a07c4" exitCode=0
Sep 16 19:15:31.715727 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.715559    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"
Sep 16 19:15:31.715727 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.715585    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln" event={"ID":"0c513fc5-dffc-4ddc-9688-41e9f46307a2","Type":"ContainerDied","Data":"0a59ca3ffbfd77dc5ea2ba81538a0342072d6fdee093f53766b024537e3a07c4"}
Sep 16 19:15:31.715727 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.715674    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln" event={"ID":"0c513fc5-dffc-4ddc-9688-41e9f46307a2","Type":"ContainerDied","Data":"9aab350a09018867065f618c0f8c2005879a0a1d6453fd9680b3893ae91202ec"}
Sep 16 19:15:31.715920 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.715750    2570 scope.go:117] "RemoveContainer" containerID="0a59ca3ffbfd77dc5ea2ba81538a0342072d6fdee093f53766b024537e3a07c4"
Sep 16 19:15:31.723850 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.723822    2570 scope.go:117] "RemoveContainer" containerID="0a59ca3ffbfd77dc5ea2ba81538a0342072d6fdee093f53766b024537e3a07c4"
Sep 16 19:15:31.724109 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:15:31.724091    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a59ca3ffbfd77dc5ea2ba81538a0342072d6fdee093f53766b024537e3a07c4\": container with ID starting with 0a59ca3ffbfd77dc5ea2ba81538a0342072d6fdee093f53766b024537e3a07c4 not found: ID does not exist" containerID="0a59ca3ffbfd77dc5ea2ba81538a0342072d6fdee093f53766b024537e3a07c4"
Sep 16 19:15:31.724154 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.724118    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a59ca3ffbfd77dc5ea2ba81538a0342072d6fdee093f53766b024537e3a07c4"} err="failed to get container status \"0a59ca3ffbfd77dc5ea2ba81538a0342072d6fdee093f53766b024537e3a07c4\": rpc error: code = NotFound desc = could not find container \"0a59ca3ffbfd77dc5ea2ba81538a0342072d6fdee093f53766b024537e3a07c4\": container with ID starting with 0a59ca3ffbfd77dc5ea2ba81538a0342072d6fdee093f53766b024537e3a07c4 not found: ID does not exist"
Sep 16 19:15:31.749495 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.749464    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"]
Sep 16 19:15:31.753476 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:31.753456    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln"]
Sep 16 19:15:32.071540 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:32.071505    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69x9hln" podUID="0c513fc5-dffc-4ddc-9688-41e9f46307a2" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.30:9003\" within 1s: context deadline exceeded"
Sep 16 19:15:32.071763 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:15:32.071625    2570 logging.go:55] [core] [Channel #209 SubChannel #210]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.30:9003", ServerName: "10.132.0.30:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.30:9003: operation was canceled"
Sep 16 19:15:32.160703 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:15:32.160656    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c513fc5-dffc-4ddc-9688-41e9f46307a2" path="/var/lib/kubelet/pods/0c513fc5-dffc-4ddc-9688-41e9f46307a2/volumes"
Sep 16 19:16:02.603010 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.602928    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"]
Sep 16 19:16:02.603387 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.603292    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c513fc5-dffc-4ddc-9688-41e9f46307a2" containerName="main"
Sep 16 19:16:02.603387 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.603310    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c513fc5-dffc-4ddc-9688-41e9f46307a2" containerName="main"
Sep 16 19:16:02.603387 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.603324    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b520559-0fc1-45ae-807c-550a7ebb92fa" containerName="storage-initializer"
Sep 16 19:16:02.603387 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.603333    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b520559-0fc1-45ae-807c-550a7ebb92fa" containerName="storage-initializer"
Sep 16 19:16:02.603387 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.603344    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b520559-0fc1-45ae-807c-550a7ebb92fa" containerName="main"
Sep 16 19:16:02.603387 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.603352    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b520559-0fc1-45ae-807c-550a7ebb92fa" containerName="main"
Sep 16 19:16:02.603591 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.603429    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c513fc5-dffc-4ddc-9688-41e9f46307a2" containerName="main"
Sep 16 19:16:02.603591 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.603441    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b520559-0fc1-45ae-807c-550a7ebb92fa" containerName="main"
Sep 16 19:16:02.606481 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.606463    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"
Sep 16 19:16:02.609866 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.609840    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-hqbzm\""
Sep 16 19:16:02.609989 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.609882    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 19:16:02.609989 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.609918    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-78qrq\""
Sep 16 19:16:02.609989 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.609957    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 19:16:02.609989 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.609920    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 19:16:02.615262 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.615238    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"]
Sep 16 19:16:02.689879 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.689854    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xt8c\" (UniqueName: \"kubernetes.io/projected/bf305393-7607-4363-b267-8d4e04ccf111-kube-api-access-9xt8c\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88\" (UID: \"bf305393-7607-4363-b267-8d4e04ccf111\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"
Sep 16 19:16:02.690003 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.689902    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf305393-7607-4363-b267-8d4e04ccf111-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88\" (UID: \"bf305393-7607-4363-b267-8d4e04ccf111\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"
Sep 16 19:16:02.790543 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.790521    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xt8c\" (UniqueName: \"kubernetes.io/projected/bf305393-7607-4363-b267-8d4e04ccf111-kube-api-access-9xt8c\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88\" (UID: \"bf305393-7607-4363-b267-8d4e04ccf111\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"
Sep 16 19:16:02.790719 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.790570    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf305393-7607-4363-b267-8d4e04ccf111-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88\" (UID: \"bf305393-7607-4363-b267-8d4e04ccf111\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"
Sep 16 19:16:02.793005 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.792982    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf305393-7607-4363-b267-8d4e04ccf111-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88\" (UID: \"bf305393-7607-4363-b267-8d4e04ccf111\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"
Sep 16 19:16:02.799544 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.799521    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xt8c\" (UniqueName: \"kubernetes.io/projected/bf305393-7607-4363-b267-8d4e04ccf111-kube-api-access-9xt8c\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88\" (UID: \"bf305393-7607-4363-b267-8d4e04ccf111\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"
Sep 16 19:16:02.916529 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:02.916468    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"
Sep 16 19:16:03.184836 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:03.184725    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"]
Sep 16 19:16:03.188085 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:16:03.188049    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf305393_7607_4363_b267_8d4e04ccf111.slice/crio-02d6b18759e96ddd1cef0901408b9a2962664dc32b80053040e34e9fd45bc6d6 WatchSource:0}: Error finding container 02d6b18759e96ddd1cef0901408b9a2962664dc32b80053040e34e9fd45bc6d6: Status 404 returned error can't find the container with id 02d6b18759e96ddd1cef0901408b9a2962664dc32b80053040e34e9fd45bc6d6
Sep 16 19:16:03.830290 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:03.830256    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88" event={"ID":"bf305393-7607-4363-b267-8d4e04ccf111","Type":"ContainerStarted","Data":"ec946bdbf023afe7befcb5aed606c7c7d74415f76180894d68b87634bd33291c"}
Sep 16 19:16:03.830714 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:03.830332    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88" event={"ID":"bf305393-7607-4363-b267-8d4e04ccf111","Type":"ContainerStarted","Data":"02d6b18759e96ddd1cef0901408b9a2962664dc32b80053040e34e9fd45bc6d6"}
Sep 16 19:16:03.830714 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:03.830375    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"
Sep 16 19:16:03.852292 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:03.852244    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88" podStartSLOduration=1.852232847 podStartE2EDuration="1.852232847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:03.850199489 +0000 UTC m=+1890.234758711" watchObservedRunningTime="2026-09-16 19:16:03.852232847 +0000 UTC m=+1890.236792069"
Sep 16 19:16:34.835309 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:16:34.835280    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"
Sep 16 19:17:31.390670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:17:31.390633    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"]
Sep 16 19:17:34.835632 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:17:34.835594    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88" podUID="bf305393-7607-4363-b267-8d4e04ccf111" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:17:44.835955 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:17:44.835912    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88" podUID="bf305393-7607-4363-b267-8d4e04ccf111" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:17:46.409223 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:17:46.409177    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88" podUID="bf305393-7607-4363-b267-8d4e04ccf111" containerName="main" containerID="cri-o://ec946bdbf023afe7befcb5aed606c7c7d74415f76180894d68b87634bd33291c" gracePeriod=45
Sep 16 19:17:54.835970 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:17:54.835930    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88" podUID="bf305393-7607-4363-b267-8d4e04ccf111" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:17:54.836443 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:17:54.836042    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"
Sep 16 19:18:04.835994 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:04.835948    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88" podUID="bf305393-7607-4363-b267-8d4e04ccf111" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:14.835451 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:14.835415    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88" podUID="bf305393-7607-4363-b267-8d4e04ccf111" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:16.646188 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:16.646165    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"
Sep 16 19:18:16.706639 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:16.706582    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf305393-7607-4363-b267-8d4e04ccf111-tls-certs\") pod \"bf305393-7607-4363-b267-8d4e04ccf111\" (UID: \"bf305393-7607-4363-b267-8d4e04ccf111\") "
Sep 16 19:18:16.706639 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:16.706619    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xt8c\" (UniqueName: \"kubernetes.io/projected/bf305393-7607-4363-b267-8d4e04ccf111-kube-api-access-9xt8c\") pod \"bf305393-7607-4363-b267-8d4e04ccf111\" (UID: \"bf305393-7607-4363-b267-8d4e04ccf111\") "
Sep 16 19:18:16.708618 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:16.708594    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf305393-7607-4363-b267-8d4e04ccf111-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf305393-7607-4363-b267-8d4e04ccf111" (UID: "bf305393-7607-4363-b267-8d4e04ccf111"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:16.708756 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:16.708734    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf305393-7607-4363-b267-8d4e04ccf111-kube-api-access-9xt8c" (OuterVolumeSpecName: "kube-api-access-9xt8c") pod "bf305393-7607-4363-b267-8d4e04ccf111" (UID: "bf305393-7607-4363-b267-8d4e04ccf111"). InnerVolumeSpecName "kube-api-access-9xt8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:16.807830 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:16.807804    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf305393-7607-4363-b267-8d4e04ccf111-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:18:16.807830 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:16.807828    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xt8c\" (UniqueName: \"kubernetes.io/projected/bf305393-7607-4363-b267-8d4e04ccf111-kube-api-access-9xt8c\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:18:17.296501 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:17.296462    2570 generic.go:356] "Generic (PLEG): container finished" podID="bf305393-7607-4363-b267-8d4e04ccf111" containerID="ec946bdbf023afe7befcb5aed606c7c7d74415f76180894d68b87634bd33291c" exitCode=0
Sep 16 19:18:17.296652 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:17.296511    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"
Sep 16 19:18:17.296652 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:17.296557    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88" event={"ID":"bf305393-7607-4363-b267-8d4e04ccf111","Type":"ContainerDied","Data":"ec946bdbf023afe7befcb5aed606c7c7d74415f76180894d68b87634bd33291c"}
Sep 16 19:18:17.296652 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:17.296610    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88" event={"ID":"bf305393-7607-4363-b267-8d4e04ccf111","Type":"ContainerDied","Data":"02d6b18759e96ddd1cef0901408b9a2962664dc32b80053040e34e9fd45bc6d6"}
Sep 16 19:18:17.296652 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:17.296635    2570 scope.go:117] "RemoveContainer" containerID="ec946bdbf023afe7befcb5aed606c7c7d74415f76180894d68b87634bd33291c"
Sep 16 19:18:17.305038 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:17.304816    2570 scope.go:117] "RemoveContainer" containerID="ec946bdbf023afe7befcb5aed606c7c7d74415f76180894d68b87634bd33291c"
Sep 16 19:18:17.305186 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:18:17.305116    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec946bdbf023afe7befcb5aed606c7c7d74415f76180894d68b87634bd33291c\": container with ID starting with ec946bdbf023afe7befcb5aed606c7c7d74415f76180894d68b87634bd33291c not found: ID does not exist" containerID="ec946bdbf023afe7befcb5aed606c7c7d74415f76180894d68b87634bd33291c"
Sep 16 19:18:17.305186 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:17.305151    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec946bdbf023afe7befcb5aed606c7c7d74415f76180894d68b87634bd33291c"} err="failed to get container status \"ec946bdbf023afe7befcb5aed606c7c7d74415f76180894d68b87634bd33291c\": rpc error: code = NotFound desc = could not find container \"ec946bdbf023afe7befcb5aed606c7c7d74415f76180894d68b87634bd33291c\": container with ID starting with ec946bdbf023afe7befcb5aed606c7c7d74415f76180894d68b87634bd33291c not found: ID does not exist"
Sep 16 19:18:17.321697 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:17.321658    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"]
Sep 16 19:18:17.324790 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:17.324768    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785zjx88"]
Sep 16 19:18:18.160934 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:18.160898    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf305393-7607-4363-b267-8d4e04ccf111" path="/var/lib/kubelet/pods/bf305393-7607-4363-b267-8d4e04ccf111/volumes"
Sep 16 19:18:49.388700 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.388661    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"]
Sep 16 19:18:49.389141 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.388963    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf305393-7607-4363-b267-8d4e04ccf111" containerName="main"
Sep 16 19:18:49.389141 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.388976    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf305393-7607-4363-b267-8d4e04ccf111" containerName="main"
Sep 16 19:18:49.389141 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.389030    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf305393-7607-4363-b267-8d4e04ccf111" containerName="main"
Sep 16 19:18:49.391730 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.391714    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.394988 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.394966    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 19:18:49.395602 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.395562    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 19:18:49.395749 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.395631    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-qkkc7\""
Sep 16 19:18:49.395749 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.395681    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 19:18:49.395870 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.395830    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 19:18:49.395928 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.395894    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:18:49.406493 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.406473    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"]
Sep 16 19:18:49.429375 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.429352    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-home\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.429510 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.429385    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.429510 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.429412    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-model-cache\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.429510 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.429435    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-dshm\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.429510 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.429460    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w8cg\" (UniqueName: \"kubernetes.io/projected/c5aef851-ed46-4801-801e-a1284bb022b7-kube-api-access-4w8cg\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.429510 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.429486    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-tmp-dir\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.429835 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.429536    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5aef851-ed46-4801-801e-a1284bb022b7-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.429835 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.429589    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5aef851-ed46-4801-801e-a1284bb022b7-tls-certs\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.530805 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.530778    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5aef851-ed46-4801-801e-a1284bb022b7-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.530927 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.530809    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5aef851-ed46-4801-801e-a1284bb022b7-tls-certs\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.530927 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.530839    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-home\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.530927 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.530856    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.530927 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.530877    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-model-cache\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.530927 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.530893    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-dshm\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.530927 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.530909    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w8cg\" (UniqueName: \"kubernetes.io/projected/c5aef851-ed46-4801-801e-a1284bb022b7-kube-api-access-4w8cg\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.530927 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.530926    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-tmp-dir\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.531269 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.531228    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-home\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.531338 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.531314    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.531432 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.531414    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-tmp-dir\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.531487 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.531422    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-model-cache\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.531751 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.531727    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5aef851-ed46-4801-801e-a1284bb022b7-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.533182 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.533160    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-dshm\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.533479 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.533462    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5aef851-ed46-4801-801e-a1284bb022b7-tls-certs\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.540006 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.539985    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w8cg\" (UniqueName: \"kubernetes.io/projected/c5aef851-ed46-4801-801e-a1284bb022b7-kube-api-access-4w8cg\") pod \"tokenizer-template-args-test-kserve-9f9c99cb6-shzqd\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.701272 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.701202    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:49.982174 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:49.982064    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"]
Sep 16 19:18:49.984770 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:18:49.984739    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5aef851_ed46_4801_801e_a1284bb022b7.slice/crio-8091d394cb7f0b7647f2e5e714bb7b6f3fa213fb4c59600fb1475179bf6e7aa4 WatchSource:0}: Error finding container 8091d394cb7f0b7647f2e5e714bb7b6f3fa213fb4c59600fb1475179bf6e7aa4: Status 404 returned error can't find the container with id 8091d394cb7f0b7647f2e5e714bb7b6f3fa213fb4c59600fb1475179bf6e7aa4
Sep 16 19:18:50.426187 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:50.426149    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd" event={"ID":"c5aef851-ed46-4801-801e-a1284bb022b7","Type":"ContainerStarted","Data":"f109d82c3ca0e7d67bc034520991142e20906343a2c45c96448ef1a1e169ad61"}
Sep 16 19:18:50.426627 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:50.426201    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd" event={"ID":"c5aef851-ed46-4801-801e-a1284bb022b7","Type":"ContainerStarted","Data":"8091d394cb7f0b7647f2e5e714bb7b6f3fa213fb4c59600fb1475179bf6e7aa4"}
Sep 16 19:18:52.434472 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:52.434390    2570 generic.go:356] "Generic (PLEG): container finished" podID="c5aef851-ed46-4801-801e-a1284bb022b7" containerID="f109d82c3ca0e7d67bc034520991142e20906343a2c45c96448ef1a1e169ad61" exitCode=0
Sep 16 19:18:52.434937 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:52.434471    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd" event={"ID":"c5aef851-ed46-4801-801e-a1284bb022b7","Type":"ContainerDied","Data":"f109d82c3ca0e7d67bc034520991142e20906343a2c45c96448ef1a1e169ad61"}
Sep 16 19:18:53.440333 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:53.440294    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd" event={"ID":"c5aef851-ed46-4801-801e-a1284bb022b7","Type":"ContainerStarted","Data":"a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632"}
Sep 16 19:18:53.462777 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:53.462729    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd" podStartSLOduration=2.356798397 podStartE2EDuration="4.462713906s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.105915509s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:53.46121119 +0000 UTC m=+2059.845770423" watchObservedRunningTime="2026-09-16 19:18:53.462713906 +0000 UTC m=+2059.847273129"
Sep 16 19:18:53.701737 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:53.701648    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:59.702065 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:59.701977    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:18:59.714333 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:18:59.714308    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:19:00.475996 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:00.475965    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:19:28.792160 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:28.792116    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"]
Sep 16 19:19:28.801176 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:19:28.801140    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:19:28.801176 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:19:28.801176 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:19:28.801176 ip-10-0-135-244 kubenswrapper[2570]:  > containerID="a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632" cmd=["/bin/sleep","15"]
Sep 16 19:19:28.801410 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:28.801193    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd" message="" err=<
Sep 16 19:19:28.801410 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:19:28.801410 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:19:28.801410 ip-10-0-135-244 kubenswrapper[2570]:  >
Sep 16 19:19:28.801410 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:19:28.801205    2570 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:19:28.801410 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:19:28.801410 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:19:28.801410 ip-10-0-135-244 kubenswrapper[2570]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd" podUID="c5aef851-ed46-4801-801e-a1284bb022b7" containerName="main" containerID="cri-o://a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632"
Sep 16 19:19:28.801410 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:28.801242    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd" podUID="c5aef851-ed46-4801-801e-a1284bb022b7" containerName="main" containerID="cri-o://a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632" gracePeriod=60
Sep 16 19:19:29.146277 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.146253    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:19:29.322185 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.322126    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-tmp-dir\") pod \"c5aef851-ed46-4801-801e-a1284bb022b7\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") "
Sep 16 19:19:29.322185 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.322168    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5aef851-ed46-4801-801e-a1284bb022b7-cabundle-cert\") pod \"c5aef851-ed46-4801-801e-a1284bb022b7\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") "
Sep 16 19:19:29.322351 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.322198    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-model-cache\") pod \"c5aef851-ed46-4801-801e-a1284bb022b7\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") "
Sep 16 19:19:29.322351 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.322228    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w8cg\" (UniqueName: \"kubernetes.io/projected/c5aef851-ed46-4801-801e-a1284bb022b7-kube-api-access-4w8cg\") pod \"c5aef851-ed46-4801-801e-a1284bb022b7\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") "
Sep 16 19:19:29.322351 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.322258    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5aef851-ed46-4801-801e-a1284bb022b7-tls-certs\") pod \"c5aef851-ed46-4801-801e-a1284bb022b7\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") "
Sep 16 19:19:29.322351 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.322298    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-home\") pod \"c5aef851-ed46-4801-801e-a1284bb022b7\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") "
Sep 16 19:19:29.322351 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.322338    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-dshm\") pod \"c5aef851-ed46-4801-801e-a1284bb022b7\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") "
Sep 16 19:19:29.322607 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.322374    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-kserve-provision-location\") pod \"c5aef851-ed46-4801-801e-a1284bb022b7\" (UID: \"c5aef851-ed46-4801-801e-a1284bb022b7\") "
Sep 16 19:19:29.322607 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.322392    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5aef851-ed46-4801-801e-a1284bb022b7" (UID: "c5aef851-ed46-4801-801e-a1284bb022b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.322607 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.322492    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5aef851-ed46-4801-801e-a1284bb022b7" (UID: "c5aef851-ed46-4801-801e-a1284bb022b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.322859 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.322622    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.322859 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.322649    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5aef851-ed46-4801-801e-a1284bb022b7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5aef851-ed46-4801-801e-a1284bb022b7" (UID: "c5aef851-ed46-4801-801e-a1284bb022b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:29.322859 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.322670    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.322859 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.322648    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-home" (OuterVolumeSpecName: "home") pod "c5aef851-ed46-4801-801e-a1284bb022b7" (UID: "c5aef851-ed46-4801-801e-a1284bb022b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.324564 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.324533    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-dshm" (OuterVolumeSpecName: "dshm") pod "c5aef851-ed46-4801-801e-a1284bb022b7" (UID: "c5aef851-ed46-4801-801e-a1284bb022b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.324766 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.324740    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5aef851-ed46-4801-801e-a1284bb022b7-kube-api-access-4w8cg" (OuterVolumeSpecName: "kube-api-access-4w8cg") pod "c5aef851-ed46-4801-801e-a1284bb022b7" (UID: "c5aef851-ed46-4801-801e-a1284bb022b7"). InnerVolumeSpecName "kube-api-access-4w8cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:29.324904 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.324882    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5aef851-ed46-4801-801e-a1284bb022b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5aef851-ed46-4801-801e-a1284bb022b7" (UID: "c5aef851-ed46-4801-801e-a1284bb022b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:29.342599 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.342573    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5aef851-ed46-4801-801e-a1284bb022b7" (UID: "c5aef851-ed46-4801-801e-a1284bb022b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.423638 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.423616    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5aef851-ed46-4801-801e-a1284bb022b7-cabundle-cert\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.423638 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.423635    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w8cg\" (UniqueName: \"kubernetes.io/projected/c5aef851-ed46-4801-801e-a1284bb022b7-kube-api-access-4w8cg\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.423777 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.423645    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5aef851-ed46-4801-801e-a1284bb022b7-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.423777 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.423654    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.423777 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.423662    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.423777 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.423670    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5aef851-ed46-4801-801e-a1284bb022b7-kserve-provision-location\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.575937 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.575867    2570 generic.go:356] "Generic (PLEG): container finished" podID="c5aef851-ed46-4801-801e-a1284bb022b7" containerID="a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632" exitCode=0
Sep 16 19:19:29.575937 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.575915    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"
Sep 16 19:19:29.576072 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.575913    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd" event={"ID":"c5aef851-ed46-4801-801e-a1284bb022b7","Type":"ContainerDied","Data":"a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632"}
Sep 16 19:19:29.576072 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.576037    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd" event={"ID":"c5aef851-ed46-4801-801e-a1284bb022b7","Type":"ContainerDied","Data":"8091d394cb7f0b7647f2e5e714bb7b6f3fa213fb4c59600fb1475179bf6e7aa4"}
Sep 16 19:19:29.576072 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.576064    2570 scope.go:117] "RemoveContainer" containerID="a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632"
Sep 16 19:19:29.585023 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.585006    2570 scope.go:117] "RemoveContainer" containerID="f109d82c3ca0e7d67bc034520991142e20906343a2c45c96448ef1a1e169ad61"
Sep 16 19:19:29.591710 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.591672    2570 scope.go:117] "RemoveContainer" containerID="a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632"
Sep 16 19:19:29.591954 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:19:29.591934    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632\": container with ID starting with a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632 not found: ID does not exist" containerID="a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632"
Sep 16 19:19:29.592003 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.591963    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632"} err="failed to get container status \"a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632\": rpc error: code = NotFound desc = could not find container \"a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632\": container with ID starting with a93b959fce6c070a6379f8ec023921c367d9d21117d937b7cdf4c4d5a9122632 not found: ID does not exist"
Sep 16 19:19:29.592003 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.591981    2570 scope.go:117] "RemoveContainer" containerID="f109d82c3ca0e7d67bc034520991142e20906343a2c45c96448ef1a1e169ad61"
Sep 16 19:19:29.592208 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:19:29.592189    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f109d82c3ca0e7d67bc034520991142e20906343a2c45c96448ef1a1e169ad61\": container with ID starting with f109d82c3ca0e7d67bc034520991142e20906343a2c45c96448ef1a1e169ad61 not found: ID does not exist" containerID="f109d82c3ca0e7d67bc034520991142e20906343a2c45c96448ef1a1e169ad61"
Sep 16 19:19:29.592264 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.592217    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f109d82c3ca0e7d67bc034520991142e20906343a2c45c96448ef1a1e169ad61"} err="failed to get container status \"f109d82c3ca0e7d67bc034520991142e20906343a2c45c96448ef1a1e169ad61\": rpc error: code = NotFound desc = could not find container \"f109d82c3ca0e7d67bc034520991142e20906343a2c45c96448ef1a1e169ad61\": container with ID starting with f109d82c3ca0e7d67bc034520991142e20906343a2c45c96448ef1a1e169ad61 not found: ID does not exist"
Sep 16 19:19:29.599207 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.599185    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"]
Sep 16 19:19:29.601240 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:29.601219    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-9f9c99cb6-shzqd"]
Sep 16 19:19:30.160619 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:19:30.160586    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5aef851-ed46-4801-801e-a1284bb022b7" path="/var/lib/kubelet/pods/c5aef851-ed46-4801-801e-a1284bb022b7/volumes"
Sep 16 19:20:41.367814 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.367783    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"]
Sep 16 19:20:41.368234 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.368054    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5aef851-ed46-4801-801e-a1284bb022b7" containerName="main"
Sep 16 19:20:41.368234 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.368065    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5aef851-ed46-4801-801e-a1284bb022b7" containerName="main"
Sep 16 19:20:41.368234 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.368076    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5aef851-ed46-4801-801e-a1284bb022b7" containerName="storage-initializer"
Sep 16 19:20:41.368234 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.368082    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5aef851-ed46-4801-801e-a1284bb022b7" containerName="storage-initializer"
Sep 16 19:20:41.368234 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.368129    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5aef851-ed46-4801-801e-a1284bb022b7" containerName="main"
Sep 16 19:20:41.370826 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.370807    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"
Sep 16 19:20:41.374006 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.373984    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-72tsx\""
Sep 16 19:20:41.374173 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.374030    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-xwg66\""
Sep 16 19:20:41.374173 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.374048    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 19:20:41.374173 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.374038    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 19:20:41.374173 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.374092    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 19:20:41.387872 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.387852    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"]
Sep 16 19:20:41.408887 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.408863    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a845a3b1-cde7-41b9-bda3-0fab19073b25-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv\" (UID: \"a845a3b1-cde7-41b9-bda3-0fab19073b25\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"
Sep 16 19:20:41.408981 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.408911    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ms99\" (UniqueName: \"kubernetes.io/projected/a845a3b1-cde7-41b9-bda3-0fab19073b25-kube-api-access-6ms99\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv\" (UID: \"a845a3b1-cde7-41b9-bda3-0fab19073b25\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"
Sep 16 19:20:41.509579 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.509545    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a845a3b1-cde7-41b9-bda3-0fab19073b25-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv\" (UID: \"a845a3b1-cde7-41b9-bda3-0fab19073b25\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"
Sep 16 19:20:41.509741 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.509599    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ms99\" (UniqueName: \"kubernetes.io/projected/a845a3b1-cde7-41b9-bda3-0fab19073b25-kube-api-access-6ms99\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv\" (UID: \"a845a3b1-cde7-41b9-bda3-0fab19073b25\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"
Sep 16 19:20:41.512017 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.511999    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a845a3b1-cde7-41b9-bda3-0fab19073b25-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv\" (UID: \"a845a3b1-cde7-41b9-bda3-0fab19073b25\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"
Sep 16 19:20:41.519180 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.519151    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ms99\" (UniqueName: \"kubernetes.io/projected/a845a3b1-cde7-41b9-bda3-0fab19073b25-kube-api-access-6ms99\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv\" (UID: \"a845a3b1-cde7-41b9-bda3-0fab19073b25\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"
Sep 16 19:20:41.680554 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.680499    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"
Sep 16 19:20:41.965941 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:41.965916    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"]
Sep 16 19:20:41.967255 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:20:41.967216    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda845a3b1_cde7_41b9_bda3_0fab19073b25.slice/crio-b82334180d4d8b9d524e31218790f401584fe3698dc25149b85b43768ae71be2 WatchSource:0}: Error finding container b82334180d4d8b9d524e31218790f401584fe3698dc25149b85b43768ae71be2: Status 404 returned error can't find the container with id b82334180d4d8b9d524e31218790f401584fe3698dc25149b85b43768ae71be2
Sep 16 19:20:42.835028 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:42.834995    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv" event={"ID":"a845a3b1-cde7-41b9-bda3-0fab19073b25","Type":"ContainerStarted","Data":"ac12c518d3f2ba14899aa0b8b74066ca3d212d98178902a3d66541ebb8ea712b"}
Sep 16 19:20:42.835391 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:42.835042    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv" event={"ID":"a845a3b1-cde7-41b9-bda3-0fab19073b25","Type":"ContainerStarted","Data":"b82334180d4d8b9d524e31218790f401584fe3698dc25149b85b43768ae71be2"}
Sep 16 19:20:42.835391 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:42.835072    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"
Sep 16 19:20:42.861217 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:20:42.861168    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv" podStartSLOduration=1.861152502 podStartE2EDuration="1.861152502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:42.860229111 +0000 UTC m=+2169.244788337" watchObservedRunningTime="2026-09-16 19:20:42.861152502 +0000 UTC m=+2169.245711758"
Sep 16 19:21:13.840322 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:21:13.840293    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"
Sep 16 19:21:39.701180 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:21:39.701147    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"]
Sep 16 19:21:43.840795 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:21:43.840757    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv" podUID="a845a3b1-cde7-41b9-bda3-0fab19073b25" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:53.839912 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:21:53.839878    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv" podUID="a845a3b1-cde7-41b9-bda3-0fab19073b25" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:54.719775 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:21:54.719735    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv" podUID="a845a3b1-cde7-41b9-bda3-0fab19073b25" containerName="main" containerID="cri-o://ac12c518d3f2ba14899aa0b8b74066ca3d212d98178902a3d66541ebb8ea712b" gracePeriod=45
Sep 16 19:22:03.840004 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:03.839897    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv" podUID="a845a3b1-cde7-41b9-bda3-0fab19073b25" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:22:03.840437 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:03.840025    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"
Sep 16 19:22:13.840140 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:13.840100    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv" podUID="a845a3b1-cde7-41b9-bda3-0fab19073b25" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:22:23.840663 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:23.840622    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv" podUID="a845a3b1-cde7-41b9-bda3-0fab19073b25" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:22:24.951035 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:24.951012    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"
Sep 16 19:22:25.019407 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.019384    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a845a3b1-cde7-41b9-bda3-0fab19073b25-tls-certs\") pod \"a845a3b1-cde7-41b9-bda3-0fab19073b25\" (UID: \"a845a3b1-cde7-41b9-bda3-0fab19073b25\") "
Sep 16 19:22:25.019511 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.019480    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ms99\" (UniqueName: \"kubernetes.io/projected/a845a3b1-cde7-41b9-bda3-0fab19073b25-kube-api-access-6ms99\") pod \"a845a3b1-cde7-41b9-bda3-0fab19073b25\" (UID: \"a845a3b1-cde7-41b9-bda3-0fab19073b25\") "
Sep 16 19:22:25.021453 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.021422    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a845a3b1-cde7-41b9-bda3-0fab19073b25-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a845a3b1-cde7-41b9-bda3-0fab19073b25" (UID: "a845a3b1-cde7-41b9-bda3-0fab19073b25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:25.021570 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.021500    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a845a3b1-cde7-41b9-bda3-0fab19073b25-kube-api-access-6ms99" (OuterVolumeSpecName: "kube-api-access-6ms99") pod "a845a3b1-cde7-41b9-bda3-0fab19073b25" (UID: "a845a3b1-cde7-41b9-bda3-0fab19073b25"). InnerVolumeSpecName "kube-api-access-6ms99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:25.119916 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.119892    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ms99\" (UniqueName: \"kubernetes.io/projected/a845a3b1-cde7-41b9-bda3-0fab19073b25-kube-api-access-6ms99\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:22:25.119916 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.119913    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a845a3b1-cde7-41b9-bda3-0fab19073b25-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:22:25.189971 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.189944    2570 generic.go:356] "Generic (PLEG): container finished" podID="a845a3b1-cde7-41b9-bda3-0fab19073b25" containerID="ac12c518d3f2ba14899aa0b8b74066ca3d212d98178902a3d66541ebb8ea712b" exitCode=0
Sep 16 19:22:25.190080 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.189985    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"
Sep 16 19:22:25.190080 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.189989    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv" event={"ID":"a845a3b1-cde7-41b9-bda3-0fab19073b25","Type":"ContainerDied","Data":"ac12c518d3f2ba14899aa0b8b74066ca3d212d98178902a3d66541ebb8ea712b"}
Sep 16 19:22:25.190158 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.190088    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv" event={"ID":"a845a3b1-cde7-41b9-bda3-0fab19073b25","Type":"ContainerDied","Data":"b82334180d4d8b9d524e31218790f401584fe3698dc25149b85b43768ae71be2"}
Sep 16 19:22:25.190158 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.190108    2570 scope.go:117] "RemoveContainer" containerID="ac12c518d3f2ba14899aa0b8b74066ca3d212d98178902a3d66541ebb8ea712b"
Sep 16 19:22:25.198461 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.198355    2570 scope.go:117] "RemoveContainer" containerID="ac12c518d3f2ba14899aa0b8b74066ca3d212d98178902a3d66541ebb8ea712b"
Sep 16 19:22:25.198730 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:22:25.198681    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac12c518d3f2ba14899aa0b8b74066ca3d212d98178902a3d66541ebb8ea712b\": container with ID starting with ac12c518d3f2ba14899aa0b8b74066ca3d212d98178902a3d66541ebb8ea712b not found: ID does not exist" containerID="ac12c518d3f2ba14899aa0b8b74066ca3d212d98178902a3d66541ebb8ea712b"
Sep 16 19:22:25.198795 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.198741    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac12c518d3f2ba14899aa0b8b74066ca3d212d98178902a3d66541ebb8ea712b"} err="failed to get container status \"ac12c518d3f2ba14899aa0b8b74066ca3d212d98178902a3d66541ebb8ea712b\": rpc error: code = NotFound desc = could not find container \"ac12c518d3f2ba14899aa0b8b74066ca3d212d98178902a3d66541ebb8ea712b\": container with ID starting with ac12c518d3f2ba14899aa0b8b74066ca3d212d98178902a3d66541ebb8ea712b not found: ID does not exist"
Sep 16 19:22:25.212081 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.212061    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"]
Sep 16 19:22:25.215954 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:25.215936    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche4kgwv"]
Sep 16 19:22:26.160227 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:22:26.160184    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a845a3b1-cde7-41b9-bda3-0fab19073b25" path="/var/lib/kubelet/pods/a845a3b1-cde7-41b9-bda3-0fab19073b25/volumes"
Sep 16 19:23:11.672123 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.672089    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"]
Sep 16 19:23:11.672576 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.672326    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a845a3b1-cde7-41b9-bda3-0fab19073b25" containerName="main"
Sep 16 19:23:11.672576 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.672338    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a845a3b1-cde7-41b9-bda3-0fab19073b25" containerName="main"
Sep 16 19:23:11.672576 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.672393    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a845a3b1-cde7-41b9-bda3-0fab19073b25" containerName="main"
Sep 16 19:23:11.674960 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.674941    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.677534 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.677515    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-j57wq\""
Sep 16 19:23:11.677625 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.677519    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:23:11.678372 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.678338    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:23:11.678465 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.678448    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 19:23:11.686200 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.686178    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"]
Sep 16 19:23:11.830483 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.830454    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fsr7\" (UniqueName: \"kubernetes.io/projected/e1ac4bb8-8593-4ae0-8748-1253efee8666-kube-api-access-5fsr7\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.830625 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.830491    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.830625 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.830511    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.830625 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.830560    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.830625 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.830596    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ac4bb8-8593-4ae0-8748-1253efee8666-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.830625 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.830622    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.931356 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.931288    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.931356 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.931318    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.931356 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.931341    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.931593 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.931363    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ac4bb8-8593-4ae0-8748-1253efee8666-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.931593 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.931379    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.931593 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.931414    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsr7\" (UniqueName: \"kubernetes.io/projected/e1ac4bb8-8593-4ae0-8748-1253efee8666-kube-api-access-5fsr7\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.931815 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.931784    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.931920 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.931847    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.931920 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.931885    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.933605 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.933587    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.934012 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.933991    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ac4bb8-8593-4ae0-8748-1253efee8666-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.951802 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.951777    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fsr7\" (UniqueName: \"kubernetes.io/projected/e1ac4bb8-8593-4ae0-8748-1253efee8666-kube-api-access-5fsr7\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:11.984764 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:11.984739    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:12.253107 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:12.253083    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"]
Sep 16 19:23:12.255575 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:23:12.255551    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ac4bb8_8593_4ae0_8748_1253efee8666.slice/crio-a5efe54ab45b7f93aa79704050111b16e9241dcbd6ba1eb4a4a563c43b200da1 WatchSource:0}: Error finding container a5efe54ab45b7f93aa79704050111b16e9241dcbd6ba1eb4a4a563c43b200da1: Status 404 returned error can't find the container with id a5efe54ab45b7f93aa79704050111b16e9241dcbd6ba1eb4a4a563c43b200da1
Sep 16 19:23:12.350208 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:12.350179    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c" event={"ID":"e1ac4bb8-8593-4ae0-8748-1253efee8666","Type":"ContainerStarted","Data":"34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2"}
Sep 16 19:23:12.350336 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:12.350234    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c" event={"ID":"e1ac4bb8-8593-4ae0-8748-1253efee8666","Type":"ContainerStarted","Data":"a5efe54ab45b7f93aa79704050111b16e9241dcbd6ba1eb4a4a563c43b200da1"}
Sep 16 19:23:12.375384 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:12.375336    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c" podStartSLOduration=1.375323047 podStartE2EDuration="1.375323047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:12.373908134 +0000 UTC m=+2318.758467369" watchObservedRunningTime="2026-09-16 19:23:12.375323047 +0000 UTC m=+2318.759882270"
Sep 16 19:23:12.985628 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:12.985597    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:21.985483 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:21.985433    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:21.997709 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:21.997670    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:23:22.395245 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:23:22.395217    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:24:26.865225 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:26.865183    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"]
Sep 16 19:24:26.868369 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:26.868348    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"
Sep 16 19:24:26.871568 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:26.871543    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-f4wq5\""
Sep 16 19:24:26.871661 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:26.871543    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 19:24:26.880771 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:26.880746    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"]
Sep 16 19:24:26.952680 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:26.952652    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm\" (UID: \"85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"
Sep 16 19:24:26.952798 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:26.952706    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z6rt\" (UniqueName: \"kubernetes.io/projected/85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942-kube-api-access-9z6rt\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm\" (UID: \"85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"
Sep 16 19:24:27.053616 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:27.053587    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z6rt\" (UniqueName: \"kubernetes.io/projected/85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942-kube-api-access-9z6rt\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm\" (UID: \"85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"
Sep 16 19:24:27.053747 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:27.053649    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm\" (UID: \"85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"
Sep 16 19:24:27.056038 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:27.056018    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm\" (UID: \"85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"
Sep 16 19:24:27.063036 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:27.063015    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z6rt\" (UniqueName: \"kubernetes.io/projected/85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942-kube-api-access-9z6rt\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm\" (UID: \"85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"
Sep 16 19:24:27.178559 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:27.178497    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"
Sep 16 19:24:27.447723 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:27.447623    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"]
Sep 16 19:24:27.450416 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:24:27.450387    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a34090_6a2d_4a1d_9d2f_d4cdb7ebf942.slice/crio-5442c8d564e3ff310764386a068799fe67d6f85dbd5a46b4dd124d88a9b54c43 WatchSource:0}: Error finding container 5442c8d564e3ff310764386a068799fe67d6f85dbd5a46b4dd124d88a9b54c43: Status 404 returned error can't find the container with id 5442c8d564e3ff310764386a068799fe67d6f85dbd5a46b4dd124d88a9b54c43
Sep 16 19:24:27.609243 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:27.609207    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm" event={"ID":"85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942","Type":"ContainerStarted","Data":"98b7869e11ec17e58647e963efa6bb61e85fdbf57cace537d54de57c25fad0ac"}
Sep 16 19:24:27.609602 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:27.609258    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm" event={"ID":"85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942","Type":"ContainerStarted","Data":"5442c8d564e3ff310764386a068799fe67d6f85dbd5a46b4dd124d88a9b54c43"}
Sep 16 19:24:27.609602 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:27.609306    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"
Sep 16 19:24:27.628210 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:27.628167    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm" podStartSLOduration=1.628153572 podStartE2EDuration="1.628153572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:27.62673462 +0000 UTC m=+2394.011293846" watchObservedRunningTime="2026-09-16 19:24:27.628153572 +0000 UTC m=+2394.012712794"
Sep 16 19:24:58.619972 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:24:58.619906    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"
Sep 16 19:25:02.717358 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:02.717326    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"]
Sep 16 19:25:08.619215 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:08.619179    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm" podUID="85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:25:17.736619 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:17.736571    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm" podUID="85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942" containerName="main" containerID="cri-o://98b7869e11ec17e58647e963efa6bb61e85fdbf57cace537d54de57c25fad0ac" gracePeriod=45
Sep 16 19:25:18.619201 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:18.619164    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm" podUID="85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:25:28.619225 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:28.619182    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm" podUID="85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:25:28.619579 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:28.619281    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"
Sep 16 19:25:38.619952 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:38.619914    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm" podUID="85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:25:47.910519 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:47.910482    2570 generic.go:356] "Generic (PLEG): container finished" podID="85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942" containerID="98b7869e11ec17e58647e963efa6bb61e85fdbf57cace537d54de57c25fad0ac" exitCode=0
Sep 16 19:25:47.910942 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:47.910545    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm" event={"ID":"85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942","Type":"ContainerDied","Data":"98b7869e11ec17e58647e963efa6bb61e85fdbf57cace537d54de57c25fad0ac"}
Sep 16 19:25:47.974491 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:47.974469    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"
Sep 16 19:25:48.033538 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:48.033511    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942-tls-certs\") pod \"85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942\" (UID: \"85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942\") "
Sep 16 19:25:48.033680 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:48.033564    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9z6rt\" (UniqueName: \"kubernetes.io/projected/85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942-kube-api-access-9z6rt\") pod \"85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942\" (UID: \"85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942\") "
Sep 16 19:25:48.035622 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:48.035591    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942" (UID: "85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:48.035737 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:48.035619    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942-kube-api-access-9z6rt" (OuterVolumeSpecName: "kube-api-access-9z6rt") pod "85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942" (UID: "85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942"). InnerVolumeSpecName "kube-api-access-9z6rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:48.134066 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:48.134042    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:25:48.134066 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:48.134065    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z6rt\" (UniqueName: \"kubernetes.io/projected/85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942-kube-api-access-9z6rt\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:25:48.915453 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:48.915420    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm" event={"ID":"85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942","Type":"ContainerDied","Data":"5442c8d564e3ff310764386a068799fe67d6f85dbd5a46b4dd124d88a9b54c43"}
Sep 16 19:25:48.915970 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:48.915456    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"
Sep 16 19:25:48.915970 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:48.915481    2570 scope.go:117] "RemoveContainer" containerID="98b7869e11ec17e58647e963efa6bb61e85fdbf57cace537d54de57c25fad0ac"
Sep 16 19:25:48.953556 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:48.953531    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"]
Sep 16 19:25:48.962481 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:48.962461    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche7vtwm"]
Sep 16 19:25:50.159752 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:50.159715    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942" path="/var/lib/kubelet/pods/85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942/volumes"
Sep 16 19:25:52.787141 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:52.787107    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"]
Sep 16 19:25:52.795309 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:25:52.795275    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:25:52.795309 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:25:52.795309 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:25:52.795309 ip-10-0-135-244 kubenswrapper[2570]:  > containerID="34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2" cmd=["/bin/sleep","15"]
Sep 16 19:25:52.795568 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:52.795322    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c" message="" err=<
Sep 16 19:25:52.795568 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:25:52.795568 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:25:52.795568 ip-10-0-135-244 kubenswrapper[2570]:  >
Sep 16 19:25:52.795568 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:25:52.795333    2570 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:25:52.795568 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:25:52.795568 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:25:52.795568 ip-10-0-135-244 kubenswrapper[2570]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c" podUID="e1ac4bb8-8593-4ae0-8748-1253efee8666" containerName="main" containerID="cri-o://34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2"
Sep 16 19:25:52.795568 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:52.795370    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c" podUID="e1ac4bb8-8593-4ae0-8748-1253efee8666" containerName="main" containerID="cri-o://34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2" gracePeriod=60
Sep 16 19:25:53.146353 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.146331    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:25:53.270693 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.270665    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ac4bb8-8593-4ae0-8748-1253efee8666-tls-certs\") pod \"e1ac4bb8-8593-4ae0-8748-1253efee8666\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") "
Sep 16 19:25:53.270838 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.270733    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-model-cache\") pod \"e1ac4bb8-8593-4ae0-8748-1253efee8666\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") "
Sep 16 19:25:53.270838 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.270772    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-tmp-dir\") pod \"e1ac4bb8-8593-4ae0-8748-1253efee8666\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") "
Sep 16 19:25:53.270838 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.270808    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-home\") pod \"e1ac4bb8-8593-4ae0-8748-1253efee8666\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") "
Sep 16 19:25:53.270962 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.270838    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-dshm\") pod \"e1ac4bb8-8593-4ae0-8748-1253efee8666\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") "
Sep 16 19:25:53.270962 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.270883    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fsr7\" (UniqueName: \"kubernetes.io/projected/e1ac4bb8-8593-4ae0-8748-1253efee8666-kube-api-access-5fsr7\") pod \"e1ac4bb8-8593-4ae0-8748-1253efee8666\" (UID: \"e1ac4bb8-8593-4ae0-8748-1253efee8666\") "
Sep 16 19:25:53.271120 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.271092    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1ac4bb8-8593-4ae0-8748-1253efee8666" (UID: "e1ac4bb8-8593-4ae0-8748-1253efee8666"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:53.271120 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.271106    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-home" (OuterVolumeSpecName: "home") pod "e1ac4bb8-8593-4ae0-8748-1253efee8666" (UID: "e1ac4bb8-8593-4ae0-8748-1253efee8666"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:53.271268 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.271126    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1ac4bb8-8593-4ae0-8748-1253efee8666" (UID: "e1ac4bb8-8593-4ae0-8748-1253efee8666"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:53.272779 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.272754    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ac4bb8-8593-4ae0-8748-1253efee8666-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1ac4bb8-8593-4ae0-8748-1253efee8666" (UID: "e1ac4bb8-8593-4ae0-8748-1253efee8666"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:53.272902 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.272878    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-dshm" (OuterVolumeSpecName: "dshm") pod "e1ac4bb8-8593-4ae0-8748-1253efee8666" (UID: "e1ac4bb8-8593-4ae0-8748-1253efee8666"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:53.272964 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.272936    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ac4bb8-8593-4ae0-8748-1253efee8666-kube-api-access-5fsr7" (OuterVolumeSpecName: "kube-api-access-5fsr7") pod "e1ac4bb8-8593-4ae0-8748-1253efee8666" (UID: "e1ac4bb8-8593-4ae0-8748-1253efee8666"). InnerVolumeSpecName "kube-api-access-5fsr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:53.372039 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.371949    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:25:53.372039 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.371979    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:25:53.372039 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.371991    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:25:53.372039 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.372002    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ac4bb8-8593-4ae0-8748-1253efee8666-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:25:53.372039 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.372013    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fsr7\" (UniqueName: \"kubernetes.io/projected/e1ac4bb8-8593-4ae0-8748-1253efee8666-kube-api-access-5fsr7\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:25:53.372039 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.372025    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ac4bb8-8593-4ae0-8748-1253efee8666-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:25:53.935024 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.934993    2570 generic.go:356] "Generic (PLEG): container finished" podID="e1ac4bb8-8593-4ae0-8748-1253efee8666" containerID="34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2" exitCode=0
Sep 16 19:25:53.935406 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.935036    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"
Sep 16 19:25:53.935406 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.935040    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c" event={"ID":"e1ac4bb8-8593-4ae0-8748-1253efee8666","Type":"ContainerDied","Data":"34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2"}
Sep 16 19:25:53.935406 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.935143    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c" event={"ID":"e1ac4bb8-8593-4ae0-8748-1253efee8666","Type":"ContainerDied","Data":"a5efe54ab45b7f93aa79704050111b16e9241dcbd6ba1eb4a4a563c43b200da1"}
Sep 16 19:25:53.935406 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.935165    2570 scope.go:117] "RemoveContainer" containerID="34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2"
Sep 16 19:25:53.943550 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.943520    2570 scope.go:117] "RemoveContainer" containerID="34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2"
Sep 16 19:25:53.943853 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:25:53.943835    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2\": container with ID starting with 34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2 not found: ID does not exist" containerID="34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2"
Sep 16 19:25:53.943942 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.943864    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2"} err="failed to get container status \"34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2\": rpc error: code = NotFound desc = could not find container \"34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2\": container with ID starting with 34c9f34963eaa9e46a08e5b68f6f80b23bd9f1a8d0e87ca7895f27a9078ad0c2 not found: ID does not exist"
Sep 16 19:25:53.959182 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.959157    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"]
Sep 16 19:25:53.963617 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:53.963597    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-tk99c"]
Sep 16 19:25:54.159661 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:25:54.159635    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1ac4bb8-8593-4ae0-8748-1253efee8666" path="/var/lib/kubelet/pods/e1ac4bb8-8593-4ae0-8748-1253efee8666/volumes"
Sep 16 19:26:51.782844 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.782809    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"]
Sep 16 19:26:51.783527 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.783499    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ac4bb8-8593-4ae0-8748-1253efee8666" containerName="main"
Sep 16 19:26:51.783719 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.783680    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ac4bb8-8593-4ae0-8748-1253efee8666" containerName="main"
Sep 16 19:26:51.783886 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.783874    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942" containerName="main"
Sep 16 19:26:51.783977 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.783967    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942" containerName="main"
Sep 16 19:26:51.784220 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.784208    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1ac4bb8-8593-4ae0-8748-1253efee8666" containerName="main"
Sep 16 19:26:51.784312 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.784303    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="85a34090-6a2d-4a1d-9d2f-d4cdb7ebf942" containerName="main"
Sep 16 19:26:51.787497 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.787473    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.792130 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.792112    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:26:51.792740 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.792719    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 19:26:51.792868 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.792767    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:26:51.793018 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.793002    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-gbshk\""
Sep 16 19:26:51.811852 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.811827    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"]
Sep 16 19:26:51.877583 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.877550    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b063728-834e-4610-bfe0-0c021da85fc2-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.877762 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.877589    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.877762 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.877614    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtpcn\" (UniqueName: \"kubernetes.io/projected/4b063728-834e-4610-bfe0-0c021da85fc2-kube-api-access-wtpcn\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.877762 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.877667    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.877762 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.877742    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.877919 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.877771    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.979100 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.979074    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.979255 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.979114    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b063728-834e-4610-bfe0-0c021da85fc2-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.979255 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.979134    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.979255 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.979154    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtpcn\" (UniqueName: \"kubernetes.io/projected/4b063728-834e-4610-bfe0-0c021da85fc2-kube-api-access-wtpcn\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.979255 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.979177    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.979255 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.979214    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.979519 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.979460    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.979587 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.979527    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.979669 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.979648    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.981390 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.981368    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.981633 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.981618    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b063728-834e-4610-bfe0-0c021da85fc2-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:51.999656 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:51.999635    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtpcn\" (UniqueName: \"kubernetes.io/projected/4b063728-834e-4610-bfe0-0c021da85fc2-kube-api-access-wtpcn\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:52.098491 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:52.098437    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:26:52.384962 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:52.384935    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"]
Sep 16 19:26:52.387322 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:26:52.387293    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b063728_834e_4610_bfe0_0c021da85fc2.slice/crio-b058f20fa9113d62d6036fe07904baee53e12de212ed3ce459562a8c0352349f WatchSource:0}: Error finding container b058f20fa9113d62d6036fe07904baee53e12de212ed3ce459562a8c0352349f: Status 404 returned error can't find the container with id b058f20fa9113d62d6036fe07904baee53e12de212ed3ce459562a8c0352349f
Sep 16 19:26:53.144280 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:53.144246    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6" event={"ID":"4b063728-834e-4610-bfe0-0c021da85fc2","Type":"ContainerStarted","Data":"983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3"}
Sep 16 19:26:53.144651 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:53.144303    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6" event={"ID":"4b063728-834e-4610-bfe0-0c021da85fc2","Type":"ContainerStarted","Data":"b058f20fa9113d62d6036fe07904baee53e12de212ed3ce459562a8c0352349f"}
Sep 16 19:26:53.169237 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:53.169191    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6" podStartSLOduration=2.169176743 podStartE2EDuration="2.169176743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:53.167899045 +0000 UTC m=+2539.552458271" watchObservedRunningTime="2026-09-16 19:26:53.169176743 +0000 UTC m=+2539.553735965"
Sep 16 19:26:54.098646 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:26:54.098614    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:27:02.099256 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:02.099206    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:27:02.111671 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:02.111646    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:27:02.187970 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:02.187938    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:27:30.599406 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:30.599370    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"]
Sep 16 19:27:30.602443 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:30.602426    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"
Sep 16 19:27:30.612385 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:30.612354    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 19:27:30.613271 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:30.613255    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-jk668\""
Sep 16 19:27:30.621417 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:30.621396    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"]
Sep 16 19:27:30.661137 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:30.661115    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6\" (UID: \"cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"
Sep 16 19:27:30.661242 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:30.661159    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6b8f\" (UniqueName: \"kubernetes.io/projected/cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa-kube-api-access-n6b8f\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6\" (UID: \"cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"
Sep 16 19:27:30.762453 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:30.762428    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6\" (UID: \"cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"
Sep 16 19:27:30.762562 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:30.762503    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6b8f\" (UniqueName: \"kubernetes.io/projected/cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa-kube-api-access-n6b8f\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6\" (UID: \"cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"
Sep 16 19:27:30.764978 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:30.764946    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6\" (UID: \"cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"
Sep 16 19:27:30.771771 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:30.771745    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6b8f\" (UniqueName: \"kubernetes.io/projected/cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa-kube-api-access-n6b8f\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6\" (UID: \"cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"
Sep 16 19:27:30.912131 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:30.912048    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"
Sep 16 19:27:31.203951 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:31.203858    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"]
Sep 16 19:27:31.207328 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:27:31.207303    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc0e5d9_c0ad_44c2_b943_48a0fb0c2afa.slice/crio-d0902bb1bb659e3a1cf9124ac3ef4723dcb4bf1478317f4a3969622339d3b97b WatchSource:0}: Error finding container d0902bb1bb659e3a1cf9124ac3ef4723dcb4bf1478317f4a3969622339d3b97b: Status 404 returned error can't find the container with id d0902bb1bb659e3a1cf9124ac3ef4723dcb4bf1478317f4a3969622339d3b97b
Sep 16 19:27:31.278936 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:31.278907    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6" event={"ID":"cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa","Type":"ContainerStarted","Data":"baa0de3c62031f68fd224df75534275700e04194198ed982adc38cded9f3539e"}
Sep 16 19:27:31.279038 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:31.278958    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6" event={"ID":"cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa","Type":"ContainerStarted","Data":"d0902bb1bb659e3a1cf9124ac3ef4723dcb4bf1478317f4a3969622339d3b97b"}
Sep 16 19:27:31.279038 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:31.278996    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"
Sep 16 19:27:31.323606 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:27:31.323558    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6" podStartSLOduration=1.32354236 podStartE2EDuration="1.32354236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:31.321940031 +0000 UTC m=+2577.706499256" watchObservedRunningTime="2026-09-16 19:27:31.32354236 +0000 UTC m=+2577.708101582"
Sep 16 19:28:02.284858 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:28:02.284784    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"
Sep 16 19:28:55.857312 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:28:55.857271    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"]
Sep 16 19:29:02.284614 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:02.284573    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6" podUID="cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:10.876958 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:10.876905    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6" podUID="cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa" containerName="main" containerID="cri-o://baa0de3c62031f68fd224df75534275700e04194198ed982adc38cded9f3539e" gracePeriod=45
Sep 16 19:29:12.284220 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:12.284177    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6" podUID="cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:22.284393 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:22.284356    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6" podUID="cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:22.284866 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:22.284477    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"
Sep 16 19:29:32.284932 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:32.284894    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6" podUID="cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:29:41.121298 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.121269    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"
Sep 16 19:29:41.178971 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.178900    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa-tls-certs\") pod \"cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa\" (UID: \"cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa\") "
Sep 16 19:29:41.178971 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.178944    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6b8f\" (UniqueName: \"kubernetes.io/projected/cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa-kube-api-access-n6b8f\") pod \"cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa\" (UID: \"cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa\") "
Sep 16 19:29:41.180979 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.180952    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa" (UID: "cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:41.181091 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.181004    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa-kube-api-access-n6b8f" (OuterVolumeSpecName: "kube-api-access-n6b8f") pod "cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa" (UID: "cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa"). InnerVolumeSpecName "kube-api-access-n6b8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:41.279749 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.279724    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:29:41.279749 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.279745    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6b8f\" (UniqueName: \"kubernetes.io/projected/cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa-kube-api-access-n6b8f\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:29:41.741157 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.741122    2570 generic.go:356] "Generic (PLEG): container finished" podID="cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa" containerID="baa0de3c62031f68fd224df75534275700e04194198ed982adc38cded9f3539e" exitCode=0
Sep 16 19:29:41.741343 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.741172    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"
Sep 16 19:29:41.741343 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.741190    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6" event={"ID":"cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa","Type":"ContainerDied","Data":"baa0de3c62031f68fd224df75534275700e04194198ed982adc38cded9f3539e"}
Sep 16 19:29:41.741343 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.741229    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6" event={"ID":"cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa","Type":"ContainerDied","Data":"d0902bb1bb659e3a1cf9124ac3ef4723dcb4bf1478317f4a3969622339d3b97b"}
Sep 16 19:29:41.741343 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.741250    2570 scope.go:117] "RemoveContainer" containerID="baa0de3c62031f68fd224df75534275700e04194198ed982adc38cded9f3539e"
Sep 16 19:29:41.749955 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.749937    2570 scope.go:117] "RemoveContainer" containerID="baa0de3c62031f68fd224df75534275700e04194198ed982adc38cded9f3539e"
Sep 16 19:29:41.750239 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:29:41.750221    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa0de3c62031f68fd224df75534275700e04194198ed982adc38cded9f3539e\": container with ID starting with baa0de3c62031f68fd224df75534275700e04194198ed982adc38cded9f3539e not found: ID does not exist" containerID="baa0de3c62031f68fd224df75534275700e04194198ed982adc38cded9f3539e"
Sep 16 19:29:41.750303 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.750249    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa0de3c62031f68fd224df75534275700e04194198ed982adc38cded9f3539e"} err="failed to get container status \"baa0de3c62031f68fd224df75534275700e04194198ed982adc38cded9f3539e\": rpc error: code = NotFound desc = could not find container \"baa0de3c62031f68fd224df75534275700e04194198ed982adc38cded9f3539e\": container with ID starting with baa0de3c62031f68fd224df75534275700e04194198ed982adc38cded9f3539e not found: ID does not exist"
Sep 16 19:29:41.762355 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.762326    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"]
Sep 16 19:29:41.765852 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:41.765833    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche8jpw6"]
Sep 16 19:29:42.160499 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.160467    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa" path="/var/lib/kubelet/pods/cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa/volumes"
Sep 16 19:29:42.386946 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.386913    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"]
Sep 16 19:29:42.395977 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:29:42.395941    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:29:42.395977 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:29:42.395977 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:29:42.395977 ip-10-0-135-244 kubenswrapper[2570]:  > containerID="983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3" cmd=["/bin/sleep","15"]
Sep 16 19:29:42.396271 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.395995    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6" message="" err=<
Sep 16 19:29:42.396271 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:29:42.396271 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:29:42.396271 ip-10-0-135-244 kubenswrapper[2570]:  >
Sep 16 19:29:42.396271 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:29:42.396012    2570 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:29:42.396271 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:29:42.396271 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 19:29:42.396271 ip-10-0-135-244 kubenswrapper[2570]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6" podUID="4b063728-834e-4610-bfe0-0c021da85fc2" containerName="main" containerID="cri-o://983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3"
Sep 16 19:29:42.396271 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.396054    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6" podUID="4b063728-834e-4610-bfe0-0c021da85fc2" containerName="main" containerID="cri-o://983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3" gracePeriod=60
Sep 16 19:29:42.739726 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.739702    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:29:42.746052 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.746021    2570 generic.go:356] "Generic (PLEG): container finished" podID="4b063728-834e-4610-bfe0-0c021da85fc2" containerID="983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3" exitCode=0
Sep 16 19:29:42.746158 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.746065    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"
Sep 16 19:29:42.746158 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.746104    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6" event={"ID":"4b063728-834e-4610-bfe0-0c021da85fc2","Type":"ContainerDied","Data":"983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3"}
Sep 16 19:29:42.746158 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.746151    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6" event={"ID":"4b063728-834e-4610-bfe0-0c021da85fc2","Type":"ContainerDied","Data":"b058f20fa9113d62d6036fe07904baee53e12de212ed3ce459562a8c0352349f"}
Sep 16 19:29:42.746316 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.746173    2570 scope.go:117] "RemoveContainer" containerID="983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3"
Sep 16 19:29:42.755127 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.755102    2570 scope.go:117] "RemoveContainer" containerID="983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3"
Sep 16 19:29:42.755434 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:29:42.755416    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3\": container with ID starting with 983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3 not found: ID does not exist" containerID="983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3"
Sep 16 19:29:42.755510 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.755445    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3"} err="failed to get container status \"983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3\": rpc error: code = NotFound desc = could not find container \"983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3\": container with ID starting with 983ce8384b87a63240783c4033d2033bfc3b608960bc8d65e62fd399d23fb7f3 not found: ID does not exist"
Sep 16 19:29:42.791432 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.791406    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-model-cache\") pod \"4b063728-834e-4610-bfe0-0c021da85fc2\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") "
Sep 16 19:29:42.791568 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.791458    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-tmp-dir\") pod \"4b063728-834e-4610-bfe0-0c021da85fc2\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") "
Sep 16 19:29:42.791568 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.791531    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtpcn\" (UniqueName: \"kubernetes.io/projected/4b063728-834e-4610-bfe0-0c021da85fc2-kube-api-access-wtpcn\") pod \"4b063728-834e-4610-bfe0-0c021da85fc2\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") "
Sep 16 19:29:42.791568 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.791557    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-dshm\") pod \"4b063728-834e-4610-bfe0-0c021da85fc2\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") "
Sep 16 19:29:42.791767 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.791584    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-home\") pod \"4b063728-834e-4610-bfe0-0c021da85fc2\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") "
Sep 16 19:29:42.791767 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.791633    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b063728-834e-4610-bfe0-0c021da85fc2-tls-certs\") pod \"4b063728-834e-4610-bfe0-0c021da85fc2\" (UID: \"4b063728-834e-4610-bfe0-0c021da85fc2\") "
Sep 16 19:29:42.791767 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.791699    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b063728-834e-4610-bfe0-0c021da85fc2" (UID: "4b063728-834e-4610-bfe0-0c021da85fc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:42.791919 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.791778    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b063728-834e-4610-bfe0-0c021da85fc2" (UID: "4b063728-834e-4610-bfe0-0c021da85fc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:42.791974 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.791930    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:29:42.791974 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.791950    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:29:42.792073 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.791980    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-home" (OuterVolumeSpecName: "home") pod "4b063728-834e-4610-bfe0-0c021da85fc2" (UID: "4b063728-834e-4610-bfe0-0c021da85fc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:42.794234 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.794211    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-dshm" (OuterVolumeSpecName: "dshm") pod "4b063728-834e-4610-bfe0-0c021da85fc2" (UID: "4b063728-834e-4610-bfe0-0c021da85fc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:42.794420 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.794397    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b063728-834e-4610-bfe0-0c021da85fc2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b063728-834e-4610-bfe0-0c021da85fc2" (UID: "4b063728-834e-4610-bfe0-0c021da85fc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:42.794478 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.794429    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b063728-834e-4610-bfe0-0c021da85fc2-kube-api-access-wtpcn" (OuterVolumeSpecName: "kube-api-access-wtpcn") pod "4b063728-834e-4610-bfe0-0c021da85fc2" (UID: "4b063728-834e-4610-bfe0-0c021da85fc2"). InnerVolumeSpecName "kube-api-access-wtpcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:42.893241 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.893213    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:29:42.893241 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.893236    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b063728-834e-4610-bfe0-0c021da85fc2-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:29:42.893241 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.893247    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtpcn\" (UniqueName: \"kubernetes.io/projected/4b063728-834e-4610-bfe0-0c021da85fc2-kube-api-access-wtpcn\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:29:42.893464 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:42.893255    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b063728-834e-4610-bfe0-0c021da85fc2-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:29:43.068148 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:43.068117    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"]
Sep 16 19:29:43.071660 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:43.071636    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969drsw6"]
Sep 16 19:29:44.162331 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:29:44.162301    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b063728-834e-4610-bfe0-0c021da85fc2" path="/var/lib/kubelet/pods/4b063728-834e-4610-bfe0-0c021da85fc2/volumes"
Sep 16 19:31:24.855900 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.855829    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"]
Sep 16 19:31:24.856386 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.856066    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa" containerName="main"
Sep 16 19:31:24.856386 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.856077    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa" containerName="main"
Sep 16 19:31:24.856386 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.856086    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b063728-834e-4610-bfe0-0c021da85fc2" containerName="main"
Sep 16 19:31:24.856386 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.856092    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b063728-834e-4610-bfe0-0c021da85fc2" containerName="main"
Sep 16 19:31:24.856386 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.856143    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc0e5d9-c0ad-44c2-b943-48a0fb0c2afa" containerName="main"
Sep 16 19:31:24.856386 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.856152    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b063728-834e-4610-bfe0-0c021da85fc2" containerName="main"
Sep 16 19:31:24.859295 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.859274    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.862062 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.862039    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 16 19:31:24.862062 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.862039    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:31:24.862354 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.862337    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-bcglm\""
Sep 16 19:31:24.862541 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.862512    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 19:31:24.862898 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.862874    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 19:31:24.863116 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.863102    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 19:31:24.874773 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.874754    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"]
Sep 16 19:31:24.897678 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.897650    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.897787 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.897680    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cck96\" (UniqueName: \"kubernetes.io/projected/98fd9102-cd14-45ca-a0ca-7a209eee6953-kube-api-access-cck96\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.897787 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.897716    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.897787 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.897768    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.897901 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.897799    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98fd9102-cd14-45ca-a0ca-7a209eee6953-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.897901 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.897825    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.897901 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.897854    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98fd9102-cd14-45ca-a0ca-7a209eee6953-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.898026 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.897924    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.999104 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.999075    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.999216 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.999112    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cck96\" (UniqueName: \"kubernetes.io/projected/98fd9102-cd14-45ca-a0ca-7a209eee6953-kube-api-access-cck96\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.999216 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.999144    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.999216 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.999175    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.999216 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.999206    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98fd9102-cd14-45ca-a0ca-7a209eee6953-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.999432 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.999281    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.999432 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.999328    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98fd9102-cd14-45ca-a0ca-7a209eee6953-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.999432 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.999379    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.999593 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.999491    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.999653 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.999630    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.999737 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.999663    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.999737 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.999681    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:24.999955 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:24.999936    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98fd9102-cd14-45ca-a0ca-7a209eee6953-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:25.001335 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:25.001317    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:25.001580 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:25.001564    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98fd9102-cd14-45ca-a0ca-7a209eee6953-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:25.007642 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:25.007625    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cck96\" (UniqueName: \"kubernetes.io/projected/98fd9102-cd14-45ca-a0ca-7a209eee6953-kube-api-access-cck96\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:25.169084 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:25.169001    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:25.447846 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:25.447809    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"]
Sep 16 19:31:25.449740 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:31:25.449714    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fd9102_cd14_45ca_a0ca_7a209eee6953.slice/crio-c649e6f1bbee0bd10708337873de67974ddb7a4fceca41b61b34be4f903a0b37 WatchSource:0}: Error finding container c649e6f1bbee0bd10708337873de67974ddb7a4fceca41b61b34be4f903a0b37: Status 404 returned error can't find the container with id c649e6f1bbee0bd10708337873de67974ddb7a4fceca41b61b34be4f903a0b37
Sep 16 19:31:26.103846 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:26.103810    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" event={"ID":"98fd9102-cd14-45ca-a0ca-7a209eee6953","Type":"ContainerStarted","Data":"0460456e8eef2b16798bc7544179df00820979925618ac2a41d1162d4b190ddb"}
Sep 16 19:31:26.104267 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:26.103870    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" event={"ID":"98fd9102-cd14-45ca-a0ca-7a209eee6953","Type":"ContainerStarted","Data":"c649e6f1bbee0bd10708337873de67974ddb7a4fceca41b61b34be4f903a0b37"}
Sep 16 19:31:29.117114 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:29.117082    2570 generic.go:356] "Generic (PLEG): container finished" podID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerID="0460456e8eef2b16798bc7544179df00820979925618ac2a41d1162d4b190ddb" exitCode=0
Sep 16 19:31:29.117479 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:29.117147    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" event={"ID":"98fd9102-cd14-45ca-a0ca-7a209eee6953","Type":"ContainerDied","Data":"0460456e8eef2b16798bc7544179df00820979925618ac2a41d1162d4b190ddb"}
Sep 16 19:31:30.123344 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:30.123258    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" event={"ID":"98fd9102-cd14-45ca-a0ca-7a209eee6953","Type":"ContainerStarted","Data":"64ae6d5c9090eef83590a593f3fd78fe0a1cad7dd750be4a31345e74eec06db8"}
Sep 16 19:31:30.148251 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:30.148205    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" podStartSLOduration=3.334125065 podStartE2EDuration="6.148190192s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.814065127s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:30.146108912 +0000 UTC m=+2816.530668134" watchObservedRunningTime="2026-09-16 19:31:30.148190192 +0000 UTC m=+2816.532749415"
Sep 16 19:31:30.169575 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:30.169547    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:35.169796 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:35.169754    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:31:35.171367 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:35.171337    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 16 19:31:45.170142 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:45.170101    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 16 19:31:55.169492 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:31:55.169450    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 16 19:32:05.170443 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:32:05.170397    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 16 19:32:15.169972 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:32:15.169930    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 16 19:32:25.169884 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:32:25.169840    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 16 19:32:35.170182 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:32:35.170140    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 16 19:32:45.179803 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:32:45.179771    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:32:45.187435 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:32:45.187402    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:33:48.001231 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:33:48.001197    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"]
Sep 16 19:34:03.022647 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.022585    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerName="main" containerID="cri-o://64ae6d5c9090eef83590a593f3fd78fe0a1cad7dd750be4a31345e74eec06db8" gracePeriod=45
Sep 16 19:34:03.180302 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.180257    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 16 19:34:03.591293 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.591264    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:34:03.671530 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.671505    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-dshm\") pod \"98fd9102-cd14-45ca-a0ca-7a209eee6953\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") "
Sep 16 19:34:03.671644 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.671534    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-tmp-dir\") pod \"98fd9102-cd14-45ca-a0ca-7a209eee6953\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") "
Sep 16 19:34:03.671644 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.671575    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-kserve-provision-location\") pod \"98fd9102-cd14-45ca-a0ca-7a209eee6953\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") "
Sep 16 19:34:03.671644 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.671608    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cck96\" (UniqueName: \"kubernetes.io/projected/98fd9102-cd14-45ca-a0ca-7a209eee6953-kube-api-access-cck96\") pod \"98fd9102-cd14-45ca-a0ca-7a209eee6953\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") "
Sep 16 19:34:03.671847 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.671655    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98fd9102-cd14-45ca-a0ca-7a209eee6953-tls-certs\") pod \"98fd9102-cd14-45ca-a0ca-7a209eee6953\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") "
Sep 16 19:34:03.671847 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.671725    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-model-cache\") pod \"98fd9102-cd14-45ca-a0ca-7a209eee6953\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") "
Sep 16 19:34:03.671847 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.671753    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-home\") pod \"98fd9102-cd14-45ca-a0ca-7a209eee6953\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") "
Sep 16 19:34:03.671847 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.671783    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98fd9102-cd14-45ca-a0ca-7a209eee6953-cabundle-cert\") pod \"98fd9102-cd14-45ca-a0ca-7a209eee6953\" (UID: \"98fd9102-cd14-45ca-a0ca-7a209eee6953\") "
Sep 16 19:34:03.672317 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.672266    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fd9102-cd14-45ca-a0ca-7a209eee6953-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "98fd9102-cd14-45ca-a0ca-7a209eee6953" (UID: "98fd9102-cd14-45ca-a0ca-7a209eee6953"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:34:03.672437 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.672338    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-home" (OuterVolumeSpecName: "home") pod "98fd9102-cd14-45ca-a0ca-7a209eee6953" (UID: "98fd9102-cd14-45ca-a0ca-7a209eee6953"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:03.672437 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.672412    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-model-cache" (OuterVolumeSpecName: "model-cache") pod "98fd9102-cd14-45ca-a0ca-7a209eee6953" (UID: "98fd9102-cd14-45ca-a0ca-7a209eee6953"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:03.673998 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.673966    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98fd9102-cd14-45ca-a0ca-7a209eee6953-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98fd9102-cd14-45ca-a0ca-7a209eee6953" (UID: "98fd9102-cd14-45ca-a0ca-7a209eee6953"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:03.673998 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.673982    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-dshm" (OuterVolumeSpecName: "dshm") pod "98fd9102-cd14-45ca-a0ca-7a209eee6953" (UID: "98fd9102-cd14-45ca-a0ca-7a209eee6953"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:03.674132 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.674054    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fd9102-cd14-45ca-a0ca-7a209eee6953-kube-api-access-cck96" (OuterVolumeSpecName: "kube-api-access-cck96") pod "98fd9102-cd14-45ca-a0ca-7a209eee6953" (UID: "98fd9102-cd14-45ca-a0ca-7a209eee6953"). InnerVolumeSpecName "kube-api-access-cck96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:03.683749 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.683721    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98fd9102-cd14-45ca-a0ca-7a209eee6953" (UID: "98fd9102-cd14-45ca-a0ca-7a209eee6953"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:03.684946 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.684924    2570 generic.go:356] "Generic (PLEG): container finished" podID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerID="64ae6d5c9090eef83590a593f3fd78fe0a1cad7dd750be4a31345e74eec06db8" exitCode=0
Sep 16 19:34:03.685023 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.684956    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" event={"ID":"98fd9102-cd14-45ca-a0ca-7a209eee6953","Type":"ContainerDied","Data":"64ae6d5c9090eef83590a593f3fd78fe0a1cad7dd750be4a31345e74eec06db8"}
Sep 16 19:34:03.685023 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.684971    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"
Sep 16 19:34:03.685023 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.684997    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls" event={"ID":"98fd9102-cd14-45ca-a0ca-7a209eee6953","Type":"ContainerDied","Data":"c649e6f1bbee0bd10708337873de67974ddb7a4fceca41b61b34be4f903a0b37"}
Sep 16 19:34:03.685023 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.685018    2570 scope.go:117] "RemoveContainer" containerID="64ae6d5c9090eef83590a593f3fd78fe0a1cad7dd750be4a31345e74eec06db8"
Sep 16 19:34:03.692460 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.692444    2570 scope.go:117] "RemoveContainer" containerID="0460456e8eef2b16798bc7544179df00820979925618ac2a41d1162d4b190ddb"
Sep 16 19:34:03.694166 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.694139    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98fd9102-cd14-45ca-a0ca-7a209eee6953" (UID: "98fd9102-cd14-45ca-a0ca-7a209eee6953"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:03.699450 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.699434    2570 scope.go:117] "RemoveContainer" containerID="64ae6d5c9090eef83590a593f3fd78fe0a1cad7dd750be4a31345e74eec06db8"
Sep 16 19:34:03.699722 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:34:03.699702    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ae6d5c9090eef83590a593f3fd78fe0a1cad7dd750be4a31345e74eec06db8\": container with ID starting with 64ae6d5c9090eef83590a593f3fd78fe0a1cad7dd750be4a31345e74eec06db8 not found: ID does not exist" containerID="64ae6d5c9090eef83590a593f3fd78fe0a1cad7dd750be4a31345e74eec06db8"
Sep 16 19:34:03.699828 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.699730    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ae6d5c9090eef83590a593f3fd78fe0a1cad7dd750be4a31345e74eec06db8"} err="failed to get container status \"64ae6d5c9090eef83590a593f3fd78fe0a1cad7dd750be4a31345e74eec06db8\": rpc error: code = NotFound desc = could not find container \"64ae6d5c9090eef83590a593f3fd78fe0a1cad7dd750be4a31345e74eec06db8\": container with ID starting with 64ae6d5c9090eef83590a593f3fd78fe0a1cad7dd750be4a31345e74eec06db8 not found: ID does not exist"
Sep 16 19:34:03.699828 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.699754    2570 scope.go:117] "RemoveContainer" containerID="0460456e8eef2b16798bc7544179df00820979925618ac2a41d1162d4b190ddb"
Sep 16 19:34:03.700061 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:34:03.700043    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0460456e8eef2b16798bc7544179df00820979925618ac2a41d1162d4b190ddb\": container with ID starting with 0460456e8eef2b16798bc7544179df00820979925618ac2a41d1162d4b190ddb not found: ID does not exist" containerID="0460456e8eef2b16798bc7544179df00820979925618ac2a41d1162d4b190ddb"
Sep 16 19:34:03.700113 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.700067    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0460456e8eef2b16798bc7544179df00820979925618ac2a41d1162d4b190ddb"} err="failed to get container status \"0460456e8eef2b16798bc7544179df00820979925618ac2a41d1162d4b190ddb\": rpc error: code = NotFound desc = could not find container \"0460456e8eef2b16798bc7544179df00820979925618ac2a41d1162d4b190ddb\": container with ID starting with 0460456e8eef2b16798bc7544179df00820979925618ac2a41d1162d4b190ddb not found: ID does not exist"
Sep 16 19:34:03.772829 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.772805    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-kserve-provision-location\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.772829 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.772827    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cck96\" (UniqueName: \"kubernetes.io/projected/98fd9102-cd14-45ca-a0ca-7a209eee6953-kube-api-access-cck96\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.772997 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.772837    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98fd9102-cd14-45ca-a0ca-7a209eee6953-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.772997 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.772846    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.772997 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.772853    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.772997 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.772860    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98fd9102-cd14-45ca-a0ca-7a209eee6953-cabundle-cert\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.772997 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.772867    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.772997 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:03.772874    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98fd9102-cd14-45ca-a0ca-7a209eee6953-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:34:04.006829 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:04.006804    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"]
Sep 16 19:34:04.011172 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:04.010964    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7594459c55-65rls"]
Sep 16 19:34:04.160795 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:04.160770    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" path="/var/lib/kubelet/pods/98fd9102-cd14-45ca-a0ca-7a209eee6953/volumes"
Sep 16 19:34:40.283706 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.283658    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"]
Sep 16 19:34:40.284235 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.284039    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerName="main"
Sep 16 19:34:40.284235 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.284057    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerName="main"
Sep 16 19:34:40.284235 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.284079    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerName="storage-initializer"
Sep 16 19:34:40.284235 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.284088    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerName="storage-initializer"
Sep 16 19:34:40.284235 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.284158    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="98fd9102-cd14-45ca-a0ca-7a209eee6953" containerName="main"
Sep 16 19:34:40.288072 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.288054    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.291041 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.291019    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 16 19:34:40.291150 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.291120    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 19:34:40.292038 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.292023    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 19:34:40.295675 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.295655    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:34:40.300008 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.299993    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 19:34:40.306811 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.306795    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-sxpkm\""
Sep 16 19:34:40.389354 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.389329    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"]
Sep 16 19:34:40.428517 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.428493    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.428607 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.428525    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.428607 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.428553    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.428607 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.428577    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.428739 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.428607    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp9xl\" (UniqueName: \"kubernetes.io/projected/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-kube-api-access-tp9xl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.428739 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.428637    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.428739 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.428704    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.428739 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.428736    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.529355 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.529331    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.529430 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.529360    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.529430 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.529389    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9xl\" (UniqueName: \"kubernetes.io/projected/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-kube-api-access-tp9xl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.529430 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.529409    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.529538 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.529444    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.529538 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.529475    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.529538 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.529505    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.529722 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.529682    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.529792 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.529776    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.529847 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.529808    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.529904 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.529859    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.529960 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.529925    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.530151 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.530132    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.531656 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.531637    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.531993 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.531977    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.540898 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.540841    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp9xl\" (UniqueName: \"kubernetes.io/projected/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-kube-api-access-tp9xl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.597588 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.597568    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:40.874774 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:40.874746    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"]
Sep 16 19:34:40.876348 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:34:40.876323    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c018199_ce1a_4d36_ba78_7a3c9c3de86d.slice/crio-b1416d16d0920cad273c22d5498d006a10bacbe569353df2336fa0a292eac0d8 WatchSource:0}: Error finding container b1416d16d0920cad273c22d5498d006a10bacbe569353df2336fa0a292eac0d8: Status 404 returned error can't find the container with id b1416d16d0920cad273c22d5498d006a10bacbe569353df2336fa0a292eac0d8
Sep 16 19:34:41.823189 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:41.823151    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" event={"ID":"7c018199-ce1a-4d36-ba78-7a3c9c3de86d","Type":"ContainerStarted","Data":"b75530ce347124f2994326646c00ce2abf227b4d99a510ba7bc85694d18c02ec"}
Sep 16 19:34:41.823574 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:41.823217    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" event={"ID":"7c018199-ce1a-4d36-ba78-7a3c9c3de86d","Type":"ContainerStarted","Data":"b1416d16d0920cad273c22d5498d006a10bacbe569353df2336fa0a292eac0d8"}
Sep 16 19:34:44.836106 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:44.836071    2570 generic.go:356] "Generic (PLEG): container finished" podID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerID="b75530ce347124f2994326646c00ce2abf227b4d99a510ba7bc85694d18c02ec" exitCode=0
Sep 16 19:34:44.836106 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:44.836108    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" event={"ID":"7c018199-ce1a-4d36-ba78-7a3c9c3de86d","Type":"ContainerDied","Data":"b75530ce347124f2994326646c00ce2abf227b4d99a510ba7bc85694d18c02ec"}
Sep 16 19:34:45.842833 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:45.842787    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" event={"ID":"7c018199-ce1a-4d36-ba78-7a3c9c3de86d","Type":"ContainerStarted","Data":"342ea92ada16922e264c5420c400784e0b8d4ca939d64ceb4bcd515449c128aa"}
Sep 16 19:34:45.889044 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:45.888980    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" podStartSLOduration=2.753367045 podStartE2EDuration="5.888962801s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.135595756s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:45.888012668 +0000 UTC m=+3012.272571906" watchObservedRunningTime="2026-09-16 19:34:45.888962801 +0000 UTC m=+3012.273522024"
Sep 16 19:34:46.598654 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:46.598606    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:50.598202 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:50.598162    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:34:50.599618 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:34:50.599589    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" podUID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 19:35:00.598631 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:35:00.598590    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" podUID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 19:35:10.598234 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:35:10.598173    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" podUID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 19:35:20.598607 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:35:20.598568    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" podUID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 19:35:30.598598 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:35:30.598550    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" podUID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 19:35:40.598296 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:35:40.598248    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" podUID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 19:35:50.598934 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:35:50.598898    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" podUID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 19:36:00.608247 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:00.608215    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:36:00.616183 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:00.616157    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:36:02.590513 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:02.590477    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"]
Sep 16 19:36:17.613425 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:17.613383    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" podUID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerName="main" containerID="cri-o://342ea92ada16922e264c5420c400784e0b8d4ca939d64ceb4bcd515449c128aa" gracePeriod=45
Sep 16 19:36:18.154073 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.154051    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:36:18.161583 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.161559    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp9xl\" (UniqueName: \"kubernetes.io/projected/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-kube-api-access-tp9xl\") pod \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") "
Sep 16 19:36:18.161711 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.161591    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-tmp-dir\") pod \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") "
Sep 16 19:36:18.161711 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.161614    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-cabundle-cert\") pod \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") "
Sep 16 19:36:18.161786 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.161770    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-home\") pod \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") "
Sep 16 19:36:18.161852 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.161836    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-tls-certs\") pod \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") "
Sep 16 19:36:18.161944 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.161918    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7c018199-ce1a-4d36-ba78-7a3c9c3de86d" (UID: "7c018199-ce1a-4d36-ba78-7a3c9c3de86d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:36:18.162096 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.162079    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c018199-ce1a-4d36-ba78-7a3c9c3de86d" (UID: "7c018199-ce1a-4d36-ba78-7a3c9c3de86d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:18.162143 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.162127    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-home" (OuterVolumeSpecName: "home") pod "7c018199-ce1a-4d36-ba78-7a3c9c3de86d" (UID: "7c018199-ce1a-4d36-ba78-7a3c9c3de86d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:18.162191 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.162156    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:36:18.162191 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.162171    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-cabundle-cert\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:36:18.163732 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.163712    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-kube-api-access-tp9xl" (OuterVolumeSpecName: "kube-api-access-tp9xl") pod "7c018199-ce1a-4d36-ba78-7a3c9c3de86d" (UID: "7c018199-ce1a-4d36-ba78-7a3c9c3de86d"). InnerVolumeSpecName "kube-api-access-tp9xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:36:18.163837 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.163813    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c018199-ce1a-4d36-ba78-7a3c9c3de86d" (UID: "7c018199-ce1a-4d36-ba78-7a3c9c3de86d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:36:18.192964 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.192940    2570 generic.go:356] "Generic (PLEG): container finished" podID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerID="342ea92ada16922e264c5420c400784e0b8d4ca939d64ceb4bcd515449c128aa" exitCode=0
Sep 16 19:36:18.193085 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.192983    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" event={"ID":"7c018199-ce1a-4d36-ba78-7a3c9c3de86d","Type":"ContainerDied","Data":"342ea92ada16922e264c5420c400784e0b8d4ca939d64ceb4bcd515449c128aa"}
Sep 16 19:36:18.193085 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.192993    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"
Sep 16 19:36:18.193085 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.193021    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6" event={"ID":"7c018199-ce1a-4d36-ba78-7a3c9c3de86d","Type":"ContainerDied","Data":"b1416d16d0920cad273c22d5498d006a10bacbe569353df2336fa0a292eac0d8"}
Sep 16 19:36:18.193085 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.193043    2570 scope.go:117] "RemoveContainer" containerID="342ea92ada16922e264c5420c400784e0b8d4ca939d64ceb4bcd515449c128aa"
Sep 16 19:36:18.200827 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.200810    2570 scope.go:117] "RemoveContainer" containerID="b75530ce347124f2994326646c00ce2abf227b4d99a510ba7bc85694d18c02ec"
Sep 16 19:36:18.208136 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.208114    2570 scope.go:117] "RemoveContainer" containerID="342ea92ada16922e264c5420c400784e0b8d4ca939d64ceb4bcd515449c128aa"
Sep 16 19:36:18.208402 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:36:18.208382    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342ea92ada16922e264c5420c400784e0b8d4ca939d64ceb4bcd515449c128aa\": container with ID starting with 342ea92ada16922e264c5420c400784e0b8d4ca939d64ceb4bcd515449c128aa not found: ID does not exist" containerID="342ea92ada16922e264c5420c400784e0b8d4ca939d64ceb4bcd515449c128aa"
Sep 16 19:36:18.208461 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.208411    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342ea92ada16922e264c5420c400784e0b8d4ca939d64ceb4bcd515449c128aa"} err="failed to get container status \"342ea92ada16922e264c5420c400784e0b8d4ca939d64ceb4bcd515449c128aa\": rpc error: code = NotFound desc = could not find container \"342ea92ada16922e264c5420c400784e0b8d4ca939d64ceb4bcd515449c128aa\": container with ID starting with 342ea92ada16922e264c5420c400784e0b8d4ca939d64ceb4bcd515449c128aa not found: ID does not exist"
Sep 16 19:36:18.208461 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.208426    2570 scope.go:117] "RemoveContainer" containerID="b75530ce347124f2994326646c00ce2abf227b4d99a510ba7bc85694d18c02ec"
Sep 16 19:36:18.208674 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:36:18.208654    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75530ce347124f2994326646c00ce2abf227b4d99a510ba7bc85694d18c02ec\": container with ID starting with b75530ce347124f2994326646c00ce2abf227b4d99a510ba7bc85694d18c02ec not found: ID does not exist" containerID="b75530ce347124f2994326646c00ce2abf227b4d99a510ba7bc85694d18c02ec"
Sep 16 19:36:18.208742 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.208699    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75530ce347124f2994326646c00ce2abf227b4d99a510ba7bc85694d18c02ec"} err="failed to get container status \"b75530ce347124f2994326646c00ce2abf227b4d99a510ba7bc85694d18c02ec\": rpc error: code = NotFound desc = could not find container \"b75530ce347124f2994326646c00ce2abf227b4d99a510ba7bc85694d18c02ec\": container with ID starting with b75530ce347124f2994326646c00ce2abf227b4d99a510ba7bc85694d18c02ec not found: ID does not exist"
Sep 16 19:36:18.263325 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.263276    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-kserve-provision-location\") pod \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") "
Sep 16 19:36:18.263325 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.263305    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-dshm\") pod \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") "
Sep 16 19:36:18.263482 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.263330    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-model-cache\") pod \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\" (UID: \"7c018199-ce1a-4d36-ba78-7a3c9c3de86d\") "
Sep 16 19:36:18.263482 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.263466    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp9xl\" (UniqueName: \"kubernetes.io/projected/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-kube-api-access-tp9xl\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:36:18.263482 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.263477    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:36:18.263722 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.263485    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:36:18.263722 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.263585    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c018199-ce1a-4d36-ba78-7a3c9c3de86d" (UID: "7c018199-ce1a-4d36-ba78-7a3c9c3de86d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:18.265121 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.265103    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-dshm" (OuterVolumeSpecName: "dshm") pod "7c018199-ce1a-4d36-ba78-7a3c9c3de86d" (UID: "7c018199-ce1a-4d36-ba78-7a3c9c3de86d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:18.283643 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.283614    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c018199-ce1a-4d36-ba78-7a3c9c3de86d" (UID: "7c018199-ce1a-4d36-ba78-7a3c9c3de86d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:36:18.363950 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.363933    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-kserve-provision-location\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:36:18.363950 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.363951    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:36:18.364083 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.363960    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c018199-ce1a-4d36-ba78-7a3c9c3de86d-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:36:18.517199 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.517149    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"]
Sep 16 19:36:18.521365 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:18.521343    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-587c7f98b8-xn6v6"]
Sep 16 19:36:20.161273 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:36:20.161232    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" path="/var/lib/kubelet/pods/7c018199-ce1a-4d36-ba78-7a3c9c3de86d/volumes"
Sep 16 19:37:12.873631 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.873550    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"]
Sep 16 19:37:12.874080 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.873816    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerName="main"
Sep 16 19:37:12.874080 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.873829    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerName="main"
Sep 16 19:37:12.874080 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.873844    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerName="storage-initializer"
Sep 16 19:37:12.874080 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.873853    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerName="storage-initializer"
Sep 16 19:37:12.874080 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.873897    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c018199-ce1a-4d36-ba78-7a3c9c3de86d" containerName="main"
Sep 16 19:37:12.876842 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.876826    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:12.880319 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.880296    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 16 19:37:12.880647 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.880630    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 16 19:37:12.880752 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.880631    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-75rzh\""
Sep 16 19:37:12.880891 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.880867    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 16 19:37:12.881011 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.880917    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 16 19:37:12.892866 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.892839    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"]
Sep 16 19:37:12.924367 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.924336    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:12.924475 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.924375    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-model-cache\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:12.924475 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.924401    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-home\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:12.924475 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.924428    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915e3931-0846-496c-982c-c5d975b7a753-tls-certs\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:12.924475 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.924456    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-dshm\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:12.924612 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.924479    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rrj4\" (UniqueName: \"kubernetes.io/projected/915e3931-0846-496c-982c-c5d975b7a753-kube-api-access-4rrj4\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:12.924612 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:12.924499    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.025641 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.025599    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-model-cache\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.025641 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.025647    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-home\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.025816 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.025763    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915e3931-0846-496c-982c-c5d975b7a753-tls-certs\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.025816 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.025807    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-dshm\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.025880 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.025846    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rrj4\" (UniqueName: \"kubernetes.io/projected/915e3931-0846-496c-982c-c5d975b7a753-kube-api-access-4rrj4\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.025914 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.025877    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.025969 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.025931    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.026080 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.026056    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-model-cache\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.026149 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.026081    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-home\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.026265 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.026246    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.026317 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.026273    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.028022 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.027999    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-dshm\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.028185 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.028168    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915e3931-0846-496c-982c-c5d975b7a753-tls-certs\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.035053 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.035034    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rrj4\" (UniqueName: \"kubernetes.io/projected/915e3931-0846-496c-982c-c5d975b7a753-kube-api-access-4rrj4\") pod \"e2e-del-blocked-svc-kserve-58b47694f5-cdcp2\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.186124 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.186053    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:13.465199 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:13.465174    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"]
Sep 16 19:37:13.467036 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:37:13.467011    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915e3931_0846_496c_982c_c5d975b7a753.slice/crio-f777c886b98d7dd79bb15f19c83f982341a76a6aa3e83071c27aa7bf92c1b6f4 WatchSource:0}: Error finding container f777c886b98d7dd79bb15f19c83f982341a76a6aa3e83071c27aa7bf92c1b6f4: Status 404 returned error can't find the container with id f777c886b98d7dd79bb15f19c83f982341a76a6aa3e83071c27aa7bf92c1b6f4
Sep 16 19:37:14.386933 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:14.386896    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2" event={"ID":"915e3931-0846-496c-982c-c5d975b7a753","Type":"ContainerStarted","Data":"eae20f2d78094b5d44fb368e5cc2cb9746c285c37100ee009c18e9716ed81085"}
Sep 16 19:37:14.387318 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:14.386951    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2" event={"ID":"915e3931-0846-496c-982c-c5d975b7a753","Type":"ContainerStarted","Data":"f777c886b98d7dd79bb15f19c83f982341a76a6aa3e83071c27aa7bf92c1b6f4"}
Sep 16 19:37:15.747506 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:15.747472    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"]
Sep 16 19:37:16.395079 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:16.395023    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2" podUID="915e3931-0846-496c-982c-c5d975b7a753" containerName="storage-initializer" containerID="cri-o://eae20f2d78094b5d44fb368e5cc2cb9746c285c37100ee009c18e9716ed81085" gracePeriod=60
Sep 16 19:37:18.781484 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.781458    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:18.881057 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.880972    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-dshm\") pod \"915e3931-0846-496c-982c-c5d975b7a753\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") "
Sep 16 19:37:18.881057 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.881028    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rrj4\" (UniqueName: \"kubernetes.io/projected/915e3931-0846-496c-982c-c5d975b7a753-kube-api-access-4rrj4\") pod \"915e3931-0846-496c-982c-c5d975b7a753\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") "
Sep 16 19:37:18.881057 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.881057    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-model-cache\") pod \"915e3931-0846-496c-982c-c5d975b7a753\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") "
Sep 16 19:37:18.881372 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.881099    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-tmp-dir\") pod \"915e3931-0846-496c-982c-c5d975b7a753\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") "
Sep 16 19:37:18.881372 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.881128    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915e3931-0846-496c-982c-c5d975b7a753-tls-certs\") pod \"915e3931-0846-496c-982c-c5d975b7a753\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") "
Sep 16 19:37:18.881372 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.881162    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-home\") pod \"915e3931-0846-496c-982c-c5d975b7a753\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") "
Sep 16 19:37:18.881372 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.881200    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-kserve-provision-location\") pod \"915e3931-0846-496c-982c-c5d975b7a753\" (UID: \"915e3931-0846-496c-982c-c5d975b7a753\") "
Sep 16 19:37:18.881372 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.881264    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-model-cache" (OuterVolumeSpecName: "model-cache") pod "915e3931-0846-496c-982c-c5d975b7a753" (UID: "915e3931-0846-496c-982c-c5d975b7a753"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:18.881372 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.881308    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "915e3931-0846-496c-982c-c5d975b7a753" (UID: "915e3931-0846-496c-982c-c5d975b7a753"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:18.881372 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.881360    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-home" (OuterVolumeSpecName: "home") pod "915e3931-0846-496c-982c-c5d975b7a753" (UID: "915e3931-0846-496c-982c-c5d975b7a753"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:18.881769 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.881500    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:37:18.881769 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.881521    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:37:18.881769 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.881535    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:37:18.883322 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.883299    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915e3931-0846-496c-982c-c5d975b7a753-kube-api-access-4rrj4" (OuterVolumeSpecName: "kube-api-access-4rrj4") pod "915e3931-0846-496c-982c-c5d975b7a753" (UID: "915e3931-0846-496c-982c-c5d975b7a753"). InnerVolumeSpecName "kube-api-access-4rrj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:37:18.883610 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.883591    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-dshm" (OuterVolumeSpecName: "dshm") pod "915e3931-0846-496c-982c-c5d975b7a753" (UID: "915e3931-0846-496c-982c-c5d975b7a753"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:18.883669 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.883589    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/915e3931-0846-496c-982c-c5d975b7a753-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "915e3931-0846-496c-982c-c5d975b7a753" (UID: "915e3931-0846-496c-982c-c5d975b7a753"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:37:18.973262 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.973223    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "915e3931-0846-496c-982c-c5d975b7a753" (UID: "915e3931-0846-496c-982c-c5d975b7a753"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:18.982475 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.982454    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-kserve-provision-location\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:37:18.982550 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.982476    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/915e3931-0846-496c-982c-c5d975b7a753-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:37:18.982550 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.982486    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rrj4\" (UniqueName: \"kubernetes.io/projected/915e3931-0846-496c-982c-c5d975b7a753-kube-api-access-4rrj4\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:37:18.982550 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:18.982496    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/915e3931-0846-496c-982c-c5d975b7a753-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:37:19.407815 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:19.407779    2570 generic.go:356] "Generic (PLEG): container finished" podID="915e3931-0846-496c-982c-c5d975b7a753" containerID="eae20f2d78094b5d44fb368e5cc2cb9746c285c37100ee009c18e9716ed81085" exitCode=0
Sep 16 19:37:19.408013 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:19.407837    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"
Sep 16 19:37:19.408013 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:19.407868    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2" event={"ID":"915e3931-0846-496c-982c-c5d975b7a753","Type":"ContainerDied","Data":"eae20f2d78094b5d44fb368e5cc2cb9746c285c37100ee009c18e9716ed81085"}
Sep 16 19:37:19.408013 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:19.407927    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2" event={"ID":"915e3931-0846-496c-982c-c5d975b7a753","Type":"ContainerDied","Data":"f777c886b98d7dd79bb15f19c83f982341a76a6aa3e83071c27aa7bf92c1b6f4"}
Sep 16 19:37:19.408013 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:19.407951    2570 scope.go:117] "RemoveContainer" containerID="eae20f2d78094b5d44fb368e5cc2cb9746c285c37100ee009c18e9716ed81085"
Sep 16 19:37:19.416930 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:19.416915    2570 scope.go:117] "RemoveContainer" containerID="eae20f2d78094b5d44fb368e5cc2cb9746c285c37100ee009c18e9716ed81085"
Sep 16 19:37:19.417176 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:37:19.417153    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae20f2d78094b5d44fb368e5cc2cb9746c285c37100ee009c18e9716ed81085\": container with ID starting with eae20f2d78094b5d44fb368e5cc2cb9746c285c37100ee009c18e9716ed81085 not found: ID does not exist" containerID="eae20f2d78094b5d44fb368e5cc2cb9746c285c37100ee009c18e9716ed81085"
Sep 16 19:37:19.417265 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:19.417181    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae20f2d78094b5d44fb368e5cc2cb9746c285c37100ee009c18e9716ed81085"} err="failed to get container status \"eae20f2d78094b5d44fb368e5cc2cb9746c285c37100ee009c18e9716ed81085\": rpc error: code = NotFound desc = could not find container \"eae20f2d78094b5d44fb368e5cc2cb9746c285c37100ee009c18e9716ed81085\": container with ID starting with eae20f2d78094b5d44fb368e5cc2cb9746c285c37100ee009c18e9716ed81085 not found: ID does not exist"
Sep 16 19:37:19.451226 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:19.451190    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"]
Sep 16 19:37:19.455703 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:19.455655    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58b47694f5-cdcp2"]
Sep 16 19:37:20.164546 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:20.164513    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="915e3931-0846-496c-982c-c5d975b7a753" path="/var/lib/kubelet/pods/915e3931-0846-496c-982c-c5d975b7a753/volumes"
Sep 16 19:37:40.453637 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.453595    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"]
Sep 16 19:37:40.454003 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.453874    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915e3931-0846-496c-982c-c5d975b7a753" containerName="storage-initializer"
Sep 16 19:37:40.454003 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.453887    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="915e3931-0846-496c-982c-c5d975b7a753" containerName="storage-initializer"
Sep 16 19:37:40.454003 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.453936    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="915e3931-0846-496c-982c-c5d975b7a753" containerName="storage-initializer"
Sep 16 19:37:40.458167 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.458150    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"
Sep 16 19:37:40.460758 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.460736    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Sep 16 19:37:40.460881 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.460826    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-6zlz8\""
Sep 16 19:37:40.461582 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.461565    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 16 19:37:40.461678 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.461588    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 16 19:37:40.466699 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.466669    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"]
Sep 16 19:37:40.629643 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.629608    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kfqx\" (UniqueName: \"kubernetes.io/projected/10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6-kube-api-access-4kfqx\") pod \"e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2\" (UID: \"10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"
Sep 16 19:37:40.629809 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.629655    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2\" (UID: \"10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"
Sep 16 19:37:40.673363 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.673327    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"]
Sep 16 19:37:40.676404 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.676387    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"
Sep 16 19:37:40.679641 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.679619    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-kserve-self-signed-certs\""
Sep 16 19:37:40.679898 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.679883    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-mk4tz\""
Sep 16 19:37:40.685919 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.685897    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"]
Sep 16 19:37:40.730024 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.729966    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kfqx\" (UniqueName: \"kubernetes.io/projected/10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6-kube-api-access-4kfqx\") pod \"e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2\" (UID: \"10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"
Sep 16 19:37:40.730024 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.730004    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2\" (UID: \"10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"
Sep 16 19:37:40.730390 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.730373    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2\" (UID: \"10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"
Sep 16 19:37:40.738184 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.738158    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kfqx\" (UniqueName: \"kubernetes.io/projected/10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6-kube-api-access-4kfqx\") pod \"e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2\" (UID: \"10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"
Sep 16 19:37:40.767913 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.767891    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"
Sep 16 19:37:40.831194 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.831167    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzptv\" (UniqueName: \"kubernetes.io/projected/7c68cde2-c1a6-4901-812c-620c4650c8fd-kube-api-access-tzptv\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf\" (UID: \"7c68cde2-c1a6-4901-812c-620c4650c8fd\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"
Sep 16 19:37:40.831384 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.831239    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf\" (UID: \"7c68cde2-c1a6-4901-812c-620c4650c8fd\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"
Sep 16 19:37:40.932263 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.932235    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf\" (UID: \"7c68cde2-c1a6-4901-812c-620c4650c8fd\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"
Sep 16 19:37:40.932402 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.932286    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzptv\" (UniqueName: \"kubernetes.io/projected/7c68cde2-c1a6-4901-812c-620c4650c8fd-kube-api-access-tzptv\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf\" (UID: \"7c68cde2-c1a6-4901-812c-620c4650c8fd\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"
Sep 16 19:37:40.934718 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.934700    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf\" (UID: \"7c68cde2-c1a6-4901-812c-620c4650c8fd\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"
Sep 16 19:37:40.940633 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.940610    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzptv\" (UniqueName: \"kubernetes.io/projected/7c68cde2-c1a6-4901-812c-620c4650c8fd-kube-api-access-tzptv\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf\" (UID: \"7c68cde2-c1a6-4901-812c-620c4650c8fd\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"
Sep 16 19:37:40.986549 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:40.986496    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"
Sep 16 19:37:41.044679 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:41.044605    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"]
Sep 16 19:37:41.047243 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:37:41.047218    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f0eb1b_fb79_4f3c_925e_f2fa8ca2cad6.slice/crio-880c47272ab0d4579aedb0fa87f3c56946e540c75b6b1afeb575213d4fdc47d5 WatchSource:0}: Error finding container 880c47272ab0d4579aedb0fa87f3c56946e540c75b6b1afeb575213d4fdc47d5: Status 404 returned error can't find the container with id 880c47272ab0d4579aedb0fa87f3c56946e540c75b6b1afeb575213d4fdc47d5
Sep 16 19:37:41.259841 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:41.259808    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"]
Sep 16 19:37:41.261409 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:37:41.261383    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c68cde2_c1a6_4901_812c_620c4650c8fd.slice/crio-bd3f42bbfc7bd6ed54f5cf8c374c5849fb69b10ce7c4a0105fc7e81f6f6e4fb5 WatchSource:0}: Error finding container bd3f42bbfc7bd6ed54f5cf8c374c5849fb69b10ce7c4a0105fc7e81f6f6e4fb5: Status 404 returned error can't find the container with id bd3f42bbfc7bd6ed54f5cf8c374c5849fb69b10ce7c4a0105fc7e81f6f6e4fb5
Sep 16 19:37:41.494866 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:41.494823    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2" event={"ID":"10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6","Type":"ContainerStarted","Data":"1fa262e0f7e91a64c53a1d2119c2a56392e38f3b80d4d62d0c4c3189c944a3f4"}
Sep 16 19:37:41.495320 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:41.494914    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2" event={"ID":"10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6","Type":"ContainerStarted","Data":"880c47272ab0d4579aedb0fa87f3c56946e540c75b6b1afeb575213d4fdc47d5"}
Sep 16 19:37:41.496404 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:41.496374    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" event={"ID":"7c68cde2-c1a6-4901-812c-620c4650c8fd","Type":"ContainerStarted","Data":"0f4b544156c8c978f9b0269a9a05137b3c5a5e24d22a0b6dcbf9245b5c88732e"}
Sep 16 19:37:41.496522 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:41.496423    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" event={"ID":"7c68cde2-c1a6-4901-812c-620c4650c8fd","Type":"ContainerStarted","Data":"bd3f42bbfc7bd6ed54f5cf8c374c5849fb69b10ce7c4a0105fc7e81f6f6e4fb5"}
Sep 16 19:37:41.496522 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:41.496443    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"
Sep 16 19:37:41.541946 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:41.541898    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" podStartSLOduration=1.5418837189999999 podStartE2EDuration="1.541883719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:41.540114879 +0000 UTC m=+3187.924674102" watchObservedRunningTime="2026-09-16 19:37:41.541883719 +0000 UTC m=+3187.926442943"
Sep 16 19:37:42.646910 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:37:42.646876    2570 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:37:42.647356 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:37:42.647283    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs podName:7c68cde2-c1a6-4901-812c-620c4650c8fd nodeName:}" failed. No retries permitted until 2026-09-16 19:37:43.147258306 +0000 UTC m=+3189.531817526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" (UID: "7c68cde2-c1a6-4901-812c-620c4650c8fd") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:37:42.684565 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:42.684530    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"]
Sep 16 19:37:42.691828 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:42.691801    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"]
Sep 16 19:37:43.152255 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:37:43.152223    2570 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:37:43.152472 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:37:43.152290    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs podName:7c68cde2-c1a6-4901-812c-620c4650c8fd nodeName:}" failed. No retries permitted until 2026-09-16 19:37:44.152276707 +0000 UTC m=+3190.536835908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" (UID: "7c68cde2-c1a6-4901-812c-620c4650c8fd") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:37:43.507792 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:43.507704    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2" podUID="10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6" containerName="storage-initializer" containerID="cri-o://1fa262e0f7e91a64c53a1d2119c2a56392e38f3b80d4d62d0c4c3189c944a3f4" gracePeriod=30
Sep 16 19:37:44.161945 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:37:44.161916    2570 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:37:44.162410 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:37:44.161981    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs podName:7c68cde2-c1a6-4901-812c-620c4650c8fd nodeName:}" failed. No retries permitted until 2026-09-16 19:37:46.161966906 +0000 UTC m=+3192.546526119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" (UID: "7c68cde2-c1a6-4901-812c-620c4650c8fd") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:37:46.151928 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.151907    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"
Sep 16 19:37:46.177078 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:37:46.177049    2570 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:37:46.177183 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:37:46.177114    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs podName:7c68cde2-c1a6-4901-812c-620c4650c8fd nodeName:}" failed. No retries permitted until 2026-09-16 19:37:50.177095179 +0000 UTC m=+3196.561654380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" (UID: "7c68cde2-c1a6-4901-812c-620c4650c8fd") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:37:46.278012 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.277986    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6-kserve-provision-location\") pod \"10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6\" (UID: \"10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6\") "
Sep 16 19:37:46.278150 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.278064    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kfqx\" (UniqueName: \"kubernetes.io/projected/10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6-kube-api-access-4kfqx\") pod \"10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6\" (UID: \"10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6\") "
Sep 16 19:37:46.280154 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.280128    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6-kube-api-access-4kfqx" (OuterVolumeSpecName: "kube-api-access-4kfqx") pod "10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6" (UID: "10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6"). InnerVolumeSpecName "kube-api-access-4kfqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:37:46.349816 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.349764    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6" (UID: "10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:46.379557 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.379532    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6-kserve-provision-location\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:37:46.379557 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.379555    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kfqx\" (UniqueName: \"kubernetes.io/projected/10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6-kube-api-access-4kfqx\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:37:46.519846 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.519817    2570 generic.go:356] "Generic (PLEG): container finished" podID="10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6" containerID="1fa262e0f7e91a64c53a1d2119c2a56392e38f3b80d4d62d0c4c3189c944a3f4" exitCode=0
Sep 16 19:37:46.519964 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.519863    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"
Sep 16 19:37:46.519964 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.519905    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2" event={"ID":"10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6","Type":"ContainerDied","Data":"1fa262e0f7e91a64c53a1d2119c2a56392e38f3b80d4d62d0c4c3189c944a3f4"}
Sep 16 19:37:46.519964 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.519958    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2" event={"ID":"10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6","Type":"ContainerDied","Data":"880c47272ab0d4579aedb0fa87f3c56946e540c75b6b1afeb575213d4fdc47d5"}
Sep 16 19:37:46.520093 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.519988    2570 scope.go:117] "RemoveContainer" containerID="1fa262e0f7e91a64c53a1d2119c2a56392e38f3b80d4d62d0c4c3189c944a3f4"
Sep 16 19:37:46.528325 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.528305    2570 scope.go:117] "RemoveContainer" containerID="1fa262e0f7e91a64c53a1d2119c2a56392e38f3b80d4d62d0c4c3189c944a3f4"
Sep 16 19:37:46.528582 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:37:46.528563    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa262e0f7e91a64c53a1d2119c2a56392e38f3b80d4d62d0c4c3189c944a3f4\": container with ID starting with 1fa262e0f7e91a64c53a1d2119c2a56392e38f3b80d4d62d0c4c3189c944a3f4 not found: ID does not exist" containerID="1fa262e0f7e91a64c53a1d2119c2a56392e38f3b80d4d62d0c4c3189c944a3f4"
Sep 16 19:37:46.528645 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.528593    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa262e0f7e91a64c53a1d2119c2a56392e38f3b80d4d62d0c4c3189c944a3f4"} err="failed to get container status \"1fa262e0f7e91a64c53a1d2119c2a56392e38f3b80d4d62d0c4c3189c944a3f4\": rpc error: code = NotFound desc = could not find container \"1fa262e0f7e91a64c53a1d2119c2a56392e38f3b80d4d62d0c4c3189c944a3f4\": container with ID starting with 1fa262e0f7e91a64c53a1d2119c2a56392e38f3b80d4d62d0c4c3189c944a3f4 not found: ID does not exist"
Sep 16 19:37:46.560073 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.560049    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"]
Sep 16 19:37:46.564877 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:46.564857    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-74b6f9f9d5-cg9h2"]
Sep 16 19:37:48.160297 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:48.160262    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6" path="/var/lib/kubelet/pods/10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6/volumes"
Sep 16 19:37:50.200890 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:37:50.200854    2570 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:37:50.201252 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:37:50.200925    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs podName:7c68cde2-c1a6-4901-812c-620c4650c8fd nodeName:}" failed. No retries permitted until 2026-09-16 19:37:58.200912142 +0000 UTC m=+3204.585471343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" (UID: "7c68cde2-c1a6-4901-812c-620c4650c8fd") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:37:51.406782 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.406750    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"]
Sep 16 19:37:51.407124 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.406990    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6" containerName="storage-initializer"
Sep 16 19:37:51.407124 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.407002    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6" containerName="storage-initializer"
Sep 16 19:37:51.407124 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.407059    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f0eb1b-fb79-4f3c-925e-f2fa8ca2cad6" containerName="storage-initializer"
Sep 16 19:37:51.409934 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.409912    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"
Sep 16 19:37:51.413802 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.413703    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 19:37:51.413802 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.413728    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-nx89k\""
Sep 16 19:37:51.413802 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.413773    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-8c7qs\""
Sep 16 19:37:51.414044 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.413775    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 19:37:51.414044 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.413706    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 19:37:51.423080 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.423059    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"]
Sep 16 19:37:51.507067 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.507043    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzmqv\" (UniqueName: \"kubernetes.io/projected/a6ac0699-ddb3-48d6-b197-92321ee5b9a8-kube-api-access-fzmqv\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh\" (UID: \"a6ac0699-ddb3-48d6-b197-92321ee5b9a8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"
Sep 16 19:37:51.507196 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.507088    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ac0699-ddb3-48d6-b197-92321ee5b9a8-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh\" (UID: \"a6ac0699-ddb3-48d6-b197-92321ee5b9a8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"
Sep 16 19:37:51.608016 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.607994    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzmqv\" (UniqueName: \"kubernetes.io/projected/a6ac0699-ddb3-48d6-b197-92321ee5b9a8-kube-api-access-fzmqv\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh\" (UID: \"a6ac0699-ddb3-48d6-b197-92321ee5b9a8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"
Sep 16 19:37:51.608134 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.608040    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ac0699-ddb3-48d6-b197-92321ee5b9a8-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh\" (UID: \"a6ac0699-ddb3-48d6-b197-92321ee5b9a8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"
Sep 16 19:37:51.610529 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.610507    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ac0699-ddb3-48d6-b197-92321ee5b9a8-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh\" (UID: \"a6ac0699-ddb3-48d6-b197-92321ee5b9a8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"
Sep 16 19:37:51.618333 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.618311    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzmqv\" (UniqueName: \"kubernetes.io/projected/a6ac0699-ddb3-48d6-b197-92321ee5b9a8-kube-api-access-fzmqv\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh\" (UID: \"a6ac0699-ddb3-48d6-b197-92321ee5b9a8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"
Sep 16 19:37:51.720056 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.719997    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"
Sep 16 19:37:51.996086 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:51.995995    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"]
Sep 16 19:37:51.998532 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:37:51.998505    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ac0699_ddb3_48d6_b197_92321ee5b9a8.slice/crio-3b74841b725b065f59b0f08de3f85a5e52c647cd16a34f438f37ac7619d81154 WatchSource:0}: Error finding container 3b74841b725b065f59b0f08de3f85a5e52c647cd16a34f438f37ac7619d81154: Status 404 returned error can't find the container with id 3b74841b725b065f59b0f08de3f85a5e52c647cd16a34f438f37ac7619d81154
Sep 16 19:37:52.546174 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:52.546136    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh" event={"ID":"a6ac0699-ddb3-48d6-b197-92321ee5b9a8","Type":"ContainerStarted","Data":"4cf6a930c30c34ca7bb8c8913b1e36a2eef63851e37cef811415b41090ad8936"}
Sep 16 19:37:52.546615 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:52.546215    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh" event={"ID":"a6ac0699-ddb3-48d6-b197-92321ee5b9a8","Type":"ContainerStarted","Data":"3b74841b725b065f59b0f08de3f85a5e52c647cd16a34f438f37ac7619d81154"}
Sep 16 19:37:52.546615 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:52.546276    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"
Sep 16 19:37:52.564947 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:52.564902    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh" podStartSLOduration=1.564888357 podStartE2EDuration="1.564888357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:52.564283218 +0000 UTC m=+3198.948842441" watchObservedRunningTime="2026-09-16 19:37:52.564888357 +0000 UTC m=+3198.949447580"
Sep 16 19:37:58.252661 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:37:58.252628    2570 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:37:58.253028 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:37:58.252720    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs podName:7c68cde2-c1a6-4901-812c-620c4650c8fd nodeName:}" failed. No retries permitted until 2026-09-16 19:38:14.252703672 +0000 UTC m=+3220.637262873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" (UID: "7c68cde2-c1a6-4901-812c-620c4650c8fd") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:37:58.529547 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:37:58.529514    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" podUID="7c68cde2-c1a6-4901-812c-620c4650c8fd" containerName="main" containerID="cri-o://0f4b544156c8c978f9b0269a9a05137b3c5a5e24d22a0b6dcbf9245b5c88732e" gracePeriod=45
Sep 16 19:38:13.510221 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:13.510179    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" podUID="7c68cde2-c1a6-4901-812c-620c4650c8fd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:14.265046 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:38:14.265016    2570 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:14.265206 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:38:14.265076    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs podName:7c68cde2-c1a6-4901-812c-620c4650c8fd nodeName:}" failed. No retries permitted until 2026-09-16 19:38:46.265062713 +0000 UTC m=+3252.649621914 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" (UID: "7c68cde2-c1a6-4901-812c-620c4650c8fd") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:38:23.509904 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:23.509864    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" podUID="7c68cde2-c1a6-4901-812c-620c4650c8fd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:23.551410 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:23.551380    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"
Sep 16 19:38:28.677372 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:28.677339    2570 generic.go:356] "Generic (PLEG): container finished" podID="7c68cde2-c1a6-4901-812c-620c4650c8fd" containerID="0f4b544156c8c978f9b0269a9a05137b3c5a5e24d22a0b6dcbf9245b5c88732e" exitCode=0
Sep 16 19:38:28.677818 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:28.677409    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" event={"ID":"7c68cde2-c1a6-4901-812c-620c4650c8fd","Type":"ContainerDied","Data":"0f4b544156c8c978f9b0269a9a05137b3c5a5e24d22a0b6dcbf9245b5c88732e"}
Sep 16 19:38:28.770580 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:28.770554    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"
Sep 16 19:38:28.874834 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:28.874765    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzptv\" (UniqueName: \"kubernetes.io/projected/7c68cde2-c1a6-4901-812c-620c4650c8fd-kube-api-access-tzptv\") pod \"7c68cde2-c1a6-4901-812c-620c4650c8fd\" (UID: \"7c68cde2-c1a6-4901-812c-620c4650c8fd\") "
Sep 16 19:38:28.874834 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:28.874807    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs\") pod \"7c68cde2-c1a6-4901-812c-620c4650c8fd\" (UID: \"7c68cde2-c1a6-4901-812c-620c4650c8fd\") "
Sep 16 19:38:28.876672 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:28.876646    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c68cde2-c1a6-4901-812c-620c4650c8fd" (UID: "7c68cde2-c1a6-4901-812c-620c4650c8fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:28.876846 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:28.876824    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c68cde2-c1a6-4901-812c-620c4650c8fd-kube-api-access-tzptv" (OuterVolumeSpecName: "kube-api-access-tzptv") pod "7c68cde2-c1a6-4901-812c-620c4650c8fd" (UID: "7c68cde2-c1a6-4901-812c-620c4650c8fd"). InnerVolumeSpecName "kube-api-access-tzptv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:28.975431 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:28.975409    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzptv\" (UniqueName: \"kubernetes.io/projected/7c68cde2-c1a6-4901-812c-620c4650c8fd-kube-api-access-tzptv\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:38:28.975431 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:28.975431    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c68cde2-c1a6-4901-812c-620c4650c8fd-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:38:29.681974 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:29.681939    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf" event={"ID":"7c68cde2-c1a6-4901-812c-620c4650c8fd","Type":"ContainerDied","Data":"bd3f42bbfc7bd6ed54f5cf8c374c5849fb69b10ce7c4a0105fc7e81f6f6e4fb5"}
Sep 16 19:38:29.681974 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:29.681973    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"
Sep 16 19:38:29.682436 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:29.682006    2570 scope.go:117] "RemoveContainer" containerID="0f4b544156c8c978f9b0269a9a05137b3c5a5e24d22a0b6dcbf9245b5c88732e"
Sep 16 19:38:29.715104 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:29.715079    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"]
Sep 16 19:38:29.721904 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:29.721878    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddjfjtf"]
Sep 16 19:38:30.160434 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:38:30.160398    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c68cde2-c1a6-4901-812c-620c4650c8fd" path="/var/lib/kubelet/pods/7c68cde2-c1a6-4901-812c-620c4650c8fd/volumes"
Sep 16 19:39:53.309778 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:39:53.309745    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"]
Sep 16 19:39:53.552132 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:39:53.552092    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh" podUID="a6ac0699-ddb3-48d6-b197-92321ee5b9a8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:40:03.551706 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:03.551608    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh" podUID="a6ac0699-ddb3-48d6-b197-92321ee5b9a8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:40:08.330665 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:08.330623    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh" podUID="a6ac0699-ddb3-48d6-b197-92321ee5b9a8" containerName="main" containerID="cri-o://4cf6a930c30c34ca7bb8c8913b1e36a2eef63851e37cef811415b41090ad8936" gracePeriod=45
Sep 16 19:40:13.552307 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:13.552271    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh" podUID="a6ac0699-ddb3-48d6-b197-92321ee5b9a8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:40:13.552661 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:13.552365    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"
Sep 16 19:40:20.340669 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.340634    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"]
Sep 16 19:40:20.341143 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.340903    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c68cde2-c1a6-4901-812c-620c4650c8fd" containerName="main"
Sep 16 19:40:20.341143 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.340914    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c68cde2-c1a6-4901-812c-620c4650c8fd" containerName="main"
Sep 16 19:40:20.341143 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.340956    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c68cde2-c1a6-4901-812c-620c4650c8fd" containerName="main"
Sep 16 19:40:20.343715 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.343672    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"
Sep 16 19:40:20.346121 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.346097    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-t72l6\""
Sep 16 19:40:20.352797 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.352774    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"]
Sep 16 19:40:20.424431 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.424411    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da92812f-596d-4576-9b53-7b29c28d25b0-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv\" (UID: \"da92812f-596d-4576-9b53-7b29c28d25b0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"
Sep 16 19:40:20.424547 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.424460    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksxfj\" (UniqueName: \"kubernetes.io/projected/da92812f-596d-4576-9b53-7b29c28d25b0-kube-api-access-ksxfj\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv\" (UID: \"da92812f-596d-4576-9b53-7b29c28d25b0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"
Sep 16 19:40:20.525283 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.525264    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da92812f-596d-4576-9b53-7b29c28d25b0-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv\" (UID: \"da92812f-596d-4576-9b53-7b29c28d25b0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"
Sep 16 19:40:20.525397 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.525313    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksxfj\" (UniqueName: \"kubernetes.io/projected/da92812f-596d-4576-9b53-7b29c28d25b0-kube-api-access-ksxfj\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv\" (UID: \"da92812f-596d-4576-9b53-7b29c28d25b0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"
Sep 16 19:40:20.527670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.527651    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da92812f-596d-4576-9b53-7b29c28d25b0-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv\" (UID: \"da92812f-596d-4576-9b53-7b29c28d25b0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"
Sep 16 19:40:20.534559 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.534539    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksxfj\" (UniqueName: \"kubernetes.io/projected/da92812f-596d-4576-9b53-7b29c28d25b0-kube-api-access-ksxfj\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv\" (UID: \"da92812f-596d-4576-9b53-7b29c28d25b0\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"
Sep 16 19:40:20.653622 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.653570    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"
Sep 16 19:40:20.923926 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:20.923900    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"]
Sep 16 19:40:20.926209 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:40:20.926180    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda92812f_596d_4576_9b53_7b29c28d25b0.slice/crio-0daf6b8109bbe60f69866c565002794a1722dccc3e265b5cd914e821a0e350b0 WatchSource:0}: Error finding container 0daf6b8109bbe60f69866c565002794a1722dccc3e265b5cd914e821a0e350b0: Status 404 returned error can't find the container with id 0daf6b8109bbe60f69866c565002794a1722dccc3e265b5cd914e821a0e350b0
Sep 16 19:40:21.058090 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:21.058057    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv" event={"ID":"da92812f-596d-4576-9b53-7b29c28d25b0","Type":"ContainerStarted","Data":"27b634637f9507414ec3c93eb7b8c41eb97fda5e00d62e8912393502ecfd6171"}
Sep 16 19:40:21.058258 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:21.058116    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv" event={"ID":"da92812f-596d-4576-9b53-7b29c28d25b0","Type":"ContainerStarted","Data":"0daf6b8109bbe60f69866c565002794a1722dccc3e265b5cd914e821a0e350b0"}
Sep 16 19:40:21.058258 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:21.058146    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"
Sep 16 19:40:21.076268 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:21.076221    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv" podStartSLOduration=1.076205604 podStartE2EDuration="1.076205604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:21.075615605 +0000 UTC m=+3347.460174830" watchObservedRunningTime="2026-09-16 19:40:21.076205604 +0000 UTC m=+3347.460764827"
Sep 16 19:40:23.552297 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:23.552253    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh" podUID="a6ac0699-ddb3-48d6-b197-92321ee5b9a8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:40:33.551841 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:33.551801    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh" podUID="a6ac0699-ddb3-48d6-b197-92321ee5b9a8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:40:38.569034 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:38.569009    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"
Sep 16 19:40:38.652815 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:38.652732    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzmqv\" (UniqueName: \"kubernetes.io/projected/a6ac0699-ddb3-48d6-b197-92321ee5b9a8-kube-api-access-fzmqv\") pod \"a6ac0699-ddb3-48d6-b197-92321ee5b9a8\" (UID: \"a6ac0699-ddb3-48d6-b197-92321ee5b9a8\") "
Sep 16 19:40:38.652815 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:38.652814    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ac0699-ddb3-48d6-b197-92321ee5b9a8-tls-certs\") pod \"a6ac0699-ddb3-48d6-b197-92321ee5b9a8\" (UID: \"a6ac0699-ddb3-48d6-b197-92321ee5b9a8\") "
Sep 16 19:40:38.655022 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:38.654992    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ac0699-ddb3-48d6-b197-92321ee5b9a8-kube-api-access-fzmqv" (OuterVolumeSpecName: "kube-api-access-fzmqv") pod "a6ac0699-ddb3-48d6-b197-92321ee5b9a8" (UID: "a6ac0699-ddb3-48d6-b197-92321ee5b9a8"). InnerVolumeSpecName "kube-api-access-fzmqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:40:38.655022 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:38.655005    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ac0699-ddb3-48d6-b197-92321ee5b9a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6ac0699-ddb3-48d6-b197-92321ee5b9a8" (UID: "a6ac0699-ddb3-48d6-b197-92321ee5b9a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:40:38.754006 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:38.753960    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ac0699-ddb3-48d6-b197-92321ee5b9a8-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:40:38.754006 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:38.753999    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzmqv\" (UniqueName: \"kubernetes.io/projected/a6ac0699-ddb3-48d6-b197-92321ee5b9a8-kube-api-access-fzmqv\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:40:39.122518 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:39.122482    2570 generic.go:356] "Generic (PLEG): container finished" podID="a6ac0699-ddb3-48d6-b197-92321ee5b9a8" containerID="4cf6a930c30c34ca7bb8c8913b1e36a2eef63851e37cef811415b41090ad8936" exitCode=0
Sep 16 19:40:39.122670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:39.122519    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"
Sep 16 19:40:39.122670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:39.122540    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh" event={"ID":"a6ac0699-ddb3-48d6-b197-92321ee5b9a8","Type":"ContainerDied","Data":"4cf6a930c30c34ca7bb8c8913b1e36a2eef63851e37cef811415b41090ad8936"}
Sep 16 19:40:39.122670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:39.122587    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh" event={"ID":"a6ac0699-ddb3-48d6-b197-92321ee5b9a8","Type":"ContainerDied","Data":"3b74841b725b065f59b0f08de3f85a5e52c647cd16a34f438f37ac7619d81154"}
Sep 16 19:40:39.122670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:39.122611    2570 scope.go:117] "RemoveContainer" containerID="4cf6a930c30c34ca7bb8c8913b1e36a2eef63851e37cef811415b41090ad8936"
Sep 16 19:40:39.130808 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:39.130787    2570 scope.go:117] "RemoveContainer" containerID="4cf6a930c30c34ca7bb8c8913b1e36a2eef63851e37cef811415b41090ad8936"
Sep 16 19:40:39.131087 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:40:39.131065    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf6a930c30c34ca7bb8c8913b1e36a2eef63851e37cef811415b41090ad8936\": container with ID starting with 4cf6a930c30c34ca7bb8c8913b1e36a2eef63851e37cef811415b41090ad8936 not found: ID does not exist" containerID="4cf6a930c30c34ca7bb8c8913b1e36a2eef63851e37cef811415b41090ad8936"
Sep 16 19:40:39.131166 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:39.131094    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf6a930c30c34ca7bb8c8913b1e36a2eef63851e37cef811415b41090ad8936"} err="failed to get container status \"4cf6a930c30c34ca7bb8c8913b1e36a2eef63851e37cef811415b41090ad8936\": rpc error: code = NotFound desc = could not find container \"4cf6a930c30c34ca7bb8c8913b1e36a2eef63851e37cef811415b41090ad8936\": container with ID starting with 4cf6a930c30c34ca7bb8c8913b1e36a2eef63851e37cef811415b41090ad8936 not found: ID does not exist"
Sep 16 19:40:39.148345 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:39.148320    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"]
Sep 16 19:40:39.152210 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:39.152190    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hwhjh"]
Sep 16 19:40:40.160452 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:40.160410    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6ac0699-ddb3-48d6-b197-92321ee5b9a8" path="/var/lib/kubelet/pods/a6ac0699-ddb3-48d6-b197-92321ee5b9a8/volumes"
Sep 16 19:40:52.063802 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:40:52.063771    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"
Sep 16 19:41:35.112869 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.112837    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"]
Sep 16 19:41:35.113259 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.113111    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6ac0699-ddb3-48d6-b197-92321ee5b9a8" containerName="main"
Sep 16 19:41:35.113259 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.113122    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6ac0699-ddb3-48d6-b197-92321ee5b9a8" containerName="main"
Sep 16 19:41:35.113259 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.113171    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6ac0699-ddb3-48d6-b197-92321ee5b9a8" containerName="main"
Sep 16 19:41:35.116269 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.116249    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"
Sep 16 19:41:35.119579 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.119558    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-mpj8t\""
Sep 16 19:41:35.120478 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.120458    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-s567f\""
Sep 16 19:41:35.120590 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.120573    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 19:41:35.120663 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.120647    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 19:41:35.126869 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.126849    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"]
Sep 16 19:41:35.127818 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.127802    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 19:41:35.216868 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.216836    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn5jk\" (UniqueName: \"kubernetes.io/projected/2e33d5a4-a06c-459c-9a49-6a321141e7eb-kube-api-access-tn5jk\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2\" (UID: \"2e33d5a4-a06c-459c-9a49-6a321141e7eb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"
Sep 16 19:41:35.217147 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.217121    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e33d5a4-a06c-459c-9a49-6a321141e7eb-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2\" (UID: \"2e33d5a4-a06c-459c-9a49-6a321141e7eb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"
Sep 16 19:41:35.318196 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.318165    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e33d5a4-a06c-459c-9a49-6a321141e7eb-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2\" (UID: \"2e33d5a4-a06c-459c-9a49-6a321141e7eb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"
Sep 16 19:41:35.318338 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.318226    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn5jk\" (UniqueName: \"kubernetes.io/projected/2e33d5a4-a06c-459c-9a49-6a321141e7eb-kube-api-access-tn5jk\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2\" (UID: \"2e33d5a4-a06c-459c-9a49-6a321141e7eb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"
Sep 16 19:41:35.320547 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.320523    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e33d5a4-a06c-459c-9a49-6a321141e7eb-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2\" (UID: \"2e33d5a4-a06c-459c-9a49-6a321141e7eb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"
Sep 16 19:41:35.326233 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.326214    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn5jk\" (UniqueName: \"kubernetes.io/projected/2e33d5a4-a06c-459c-9a49-6a321141e7eb-kube-api-access-tn5jk\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2\" (UID: \"2e33d5a4-a06c-459c-9a49-6a321141e7eb\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"
Sep 16 19:41:35.425538 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.425478    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"
Sep 16 19:41:35.706943 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:35.706915    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"]
Sep 16 19:41:35.707637 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:41:35.707612    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e33d5a4_a06c_459c_9a49_6a321141e7eb.slice/crio-07cec918798fd835edd82676a06513cd6d517f634af83a99c00544752ce9efa6 WatchSource:0}: Error finding container 07cec918798fd835edd82676a06513cd6d517f634af83a99c00544752ce9efa6: Status 404 returned error can't find the container with id 07cec918798fd835edd82676a06513cd6d517f634af83a99c00544752ce9efa6
Sep 16 19:41:36.320658 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:36.320623    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2" event={"ID":"2e33d5a4-a06c-459c-9a49-6a321141e7eb","Type":"ContainerStarted","Data":"a0c028a91f1b86e917236064ec00bcd7589756ff766d36d1fcc60219e9c415a7"}
Sep 16 19:41:36.321035 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:36.320673    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2" event={"ID":"2e33d5a4-a06c-459c-9a49-6a321141e7eb","Type":"ContainerStarted","Data":"07cec918798fd835edd82676a06513cd6d517f634af83a99c00544752ce9efa6"}
Sep 16 19:41:36.321035 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:36.320766    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"
Sep 16 19:41:36.338993 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:41:36.338948    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2" podStartSLOduration=1.3389360240000001 podStartE2EDuration="1.338936024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:36.337960952 +0000 UTC m=+3422.722520175" watchObservedRunningTime="2026-09-16 19:41:36.338936024 +0000 UTC m=+3422.723495245"
Sep 16 19:42:07.326604 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:42:07.326569    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"
Sep 16 19:42:22.291585 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:42:22.291547    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"]
Sep 16 19:42:32.063891 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:42:32.063856    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv" podUID="da92812f-596d-4576-9b53-7b29c28d25b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:37.309899 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:42:37.309856    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv" podUID="da92812f-596d-4576-9b53-7b29c28d25b0" containerName="main" containerID="cri-o://27b634637f9507414ec3c93eb7b8c41eb97fda5e00d62e8912393502ecfd6171" gracePeriod=45
Sep 16 19:42:42.063970 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:42:42.063931    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv" podUID="da92812f-596d-4576-9b53-7b29c28d25b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:52.063610 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:42:52.063567    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv" podUID="da92812f-596d-4576-9b53-7b29c28d25b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:52.064003 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:42:52.063720    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"
Sep 16 19:43:02.064102 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:02.064010    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv" podUID="da92812f-596d-4576-9b53-7b29c28d25b0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:43:07.546707 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.546657    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"
Sep 16 19:43:07.639415 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.639328    2570 generic.go:356] "Generic (PLEG): container finished" podID="da92812f-596d-4576-9b53-7b29c28d25b0" containerID="27b634637f9507414ec3c93eb7b8c41eb97fda5e00d62e8912393502ecfd6171" exitCode=0
Sep 16 19:43:07.639415 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.639371    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"
Sep 16 19:43:07.639415 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.639394    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv" event={"ID":"da92812f-596d-4576-9b53-7b29c28d25b0","Type":"ContainerDied","Data":"27b634637f9507414ec3c93eb7b8c41eb97fda5e00d62e8912393502ecfd6171"}
Sep 16 19:43:07.639640 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.639434    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv" event={"ID":"da92812f-596d-4576-9b53-7b29c28d25b0","Type":"ContainerDied","Data":"0daf6b8109bbe60f69866c565002794a1722dccc3e265b5cd914e821a0e350b0"}
Sep 16 19:43:07.639640 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.639455    2570 scope.go:117] "RemoveContainer" containerID="27b634637f9507414ec3c93eb7b8c41eb97fda5e00d62e8912393502ecfd6171"
Sep 16 19:43:07.647095 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.647076    2570 scope.go:117] "RemoveContainer" containerID="27b634637f9507414ec3c93eb7b8c41eb97fda5e00d62e8912393502ecfd6171"
Sep 16 19:43:07.647374 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:43:07.647351    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b634637f9507414ec3c93eb7b8c41eb97fda5e00d62e8912393502ecfd6171\": container with ID starting with 27b634637f9507414ec3c93eb7b8c41eb97fda5e00d62e8912393502ecfd6171 not found: ID does not exist" containerID="27b634637f9507414ec3c93eb7b8c41eb97fda5e00d62e8912393502ecfd6171"
Sep 16 19:43:07.647467 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.647379    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b634637f9507414ec3c93eb7b8c41eb97fda5e00d62e8912393502ecfd6171"} err="failed to get container status \"27b634637f9507414ec3c93eb7b8c41eb97fda5e00d62e8912393502ecfd6171\": rpc error: code = NotFound desc = could not find container \"27b634637f9507414ec3c93eb7b8c41eb97fda5e00d62e8912393502ecfd6171\": container with ID starting with 27b634637f9507414ec3c93eb7b8c41eb97fda5e00d62e8912393502ecfd6171 not found: ID does not exist"
Sep 16 19:43:07.657562 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.657546    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da92812f-596d-4576-9b53-7b29c28d25b0-tls-certs\") pod \"da92812f-596d-4576-9b53-7b29c28d25b0\" (UID: \"da92812f-596d-4576-9b53-7b29c28d25b0\") "
Sep 16 19:43:07.657640 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.657572    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksxfj\" (UniqueName: \"kubernetes.io/projected/da92812f-596d-4576-9b53-7b29c28d25b0-kube-api-access-ksxfj\") pod \"da92812f-596d-4576-9b53-7b29c28d25b0\" (UID: \"da92812f-596d-4576-9b53-7b29c28d25b0\") "
Sep 16 19:43:07.659573 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.659551    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da92812f-596d-4576-9b53-7b29c28d25b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da92812f-596d-4576-9b53-7b29c28d25b0" (UID: "da92812f-596d-4576-9b53-7b29c28d25b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:07.659656 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.659620    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da92812f-596d-4576-9b53-7b29c28d25b0-kube-api-access-ksxfj" (OuterVolumeSpecName: "kube-api-access-ksxfj") pod "da92812f-596d-4576-9b53-7b29c28d25b0" (UID: "da92812f-596d-4576-9b53-7b29c28d25b0"). InnerVolumeSpecName "kube-api-access-ksxfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:07.758631 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.758598    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da92812f-596d-4576-9b53-7b29c28d25b0-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:43:07.758631 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.758625    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksxfj\" (UniqueName: \"kubernetes.io/projected/da92812f-596d-4576-9b53-7b29c28d25b0-kube-api-access-ksxfj\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:43:07.960345 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.960318    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"]
Sep 16 19:43:07.966758 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:07.966733    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-8zxtv"]
Sep 16 19:43:08.160362 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:08.160332    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da92812f-596d-4576-9b53-7b29c28d25b0" path="/var/lib/kubelet/pods/da92812f-596d-4576-9b53-7b29c28d25b0/volumes"
Sep 16 19:43:50.051482 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:50.051445    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"]
Sep 16 19:43:57.326594 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:43:57.326554    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2" podUID="2e33d5a4-a06c-459c-9a49-6a321141e7eb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:44:05.069649 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:05.069604    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2" podUID="2e33d5a4-a06c-459c-9a49-6a321141e7eb" containerName="main" containerID="cri-o://a0c028a91f1b86e917236064ec00bcd7589756ff766d36d1fcc60219e9c415a7" gracePeriod=45
Sep 16 19:44:07.326237 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:07.326198    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2" podUID="2e33d5a4-a06c-459c-9a49-6a321141e7eb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:44:17.325815 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:17.325776    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2" podUID="2e33d5a4-a06c-459c-9a49-6a321141e7eb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:44:17.326191 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:17.325882    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"
Sep 16 19:44:27.326399 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:27.326320    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2" podUID="2e33d5a4-a06c-459c-9a49-6a321141e7eb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:44:35.315669 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.315644    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"
Sep 16 19:44:35.474583 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.474517    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn5jk\" (UniqueName: \"kubernetes.io/projected/2e33d5a4-a06c-459c-9a49-6a321141e7eb-kube-api-access-tn5jk\") pod \"2e33d5a4-a06c-459c-9a49-6a321141e7eb\" (UID: \"2e33d5a4-a06c-459c-9a49-6a321141e7eb\") "
Sep 16 19:44:35.474583 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.474559    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e33d5a4-a06c-459c-9a49-6a321141e7eb-tls-certs\") pod \"2e33d5a4-a06c-459c-9a49-6a321141e7eb\" (UID: \"2e33d5a4-a06c-459c-9a49-6a321141e7eb\") "
Sep 16 19:44:35.476725 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.476677    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e33d5a4-a06c-459c-9a49-6a321141e7eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e33d5a4-a06c-459c-9a49-6a321141e7eb" (UID: "2e33d5a4-a06c-459c-9a49-6a321141e7eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:44:35.476837 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.476808    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e33d5a4-a06c-459c-9a49-6a321141e7eb-kube-api-access-tn5jk" (OuterVolumeSpecName: "kube-api-access-tn5jk") pod "2e33d5a4-a06c-459c-9a49-6a321141e7eb" (UID: "2e33d5a4-a06c-459c-9a49-6a321141e7eb"). InnerVolumeSpecName "kube-api-access-tn5jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:44:35.575949 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.575923    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn5jk\" (UniqueName: \"kubernetes.io/projected/2e33d5a4-a06c-459c-9a49-6a321141e7eb-kube-api-access-tn5jk\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:44:35.575949 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.575947    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e33d5a4-a06c-459c-9a49-6a321141e7eb-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:44:35.944420 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.944393    2570 generic.go:356] "Generic (PLEG): container finished" podID="2e33d5a4-a06c-459c-9a49-6a321141e7eb" containerID="a0c028a91f1b86e917236064ec00bcd7589756ff766d36d1fcc60219e9c415a7" exitCode=0
Sep 16 19:44:35.944678 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.944455    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"
Sep 16 19:44:35.944678 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.944483    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2" event={"ID":"2e33d5a4-a06c-459c-9a49-6a321141e7eb","Type":"ContainerDied","Data":"a0c028a91f1b86e917236064ec00bcd7589756ff766d36d1fcc60219e9c415a7"}
Sep 16 19:44:35.944678 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.944541    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2" event={"ID":"2e33d5a4-a06c-459c-9a49-6a321141e7eb","Type":"ContainerDied","Data":"07cec918798fd835edd82676a06513cd6d517f634af83a99c00544752ce9efa6"}
Sep 16 19:44:35.944678 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.944570    2570 scope.go:117] "RemoveContainer" containerID="a0c028a91f1b86e917236064ec00bcd7589756ff766d36d1fcc60219e9c415a7"
Sep 16 19:44:35.954423 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.954403    2570 scope.go:117] "RemoveContainer" containerID="a0c028a91f1b86e917236064ec00bcd7589756ff766d36d1fcc60219e9c415a7"
Sep 16 19:44:35.954725 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:44:35.954705    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c028a91f1b86e917236064ec00bcd7589756ff766d36d1fcc60219e9c415a7\": container with ID starting with a0c028a91f1b86e917236064ec00bcd7589756ff766d36d1fcc60219e9c415a7 not found: ID does not exist" containerID="a0c028a91f1b86e917236064ec00bcd7589756ff766d36d1fcc60219e9c415a7"
Sep 16 19:44:35.954780 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.954731    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c028a91f1b86e917236064ec00bcd7589756ff766d36d1fcc60219e9c415a7"} err="failed to get container status \"a0c028a91f1b86e917236064ec00bcd7589756ff766d36d1fcc60219e9c415a7\": rpc error: code = NotFound desc = could not find container \"a0c028a91f1b86e917236064ec00bcd7589756ff766d36d1fcc60219e9c415a7\": container with ID starting with a0c028a91f1b86e917236064ec00bcd7589756ff766d36d1fcc60219e9c415a7 not found: ID does not exist"
Sep 16 19:44:35.966552 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.966528    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"]
Sep 16 19:44:35.971078 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:35.971055    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh6kz2"]
Sep 16 19:44:36.159967 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:36.159942    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e33d5a4-a06c-459c-9a49-6a321141e7eb" path="/var/lib/kubelet/pods/2e33d5a4-a06c-459c-9a49-6a321141e7eb/volumes"
Sep 16 19:44:51.697765 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.697731    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"]
Sep 16 19:44:51.698185 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.698031    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da92812f-596d-4576-9b53-7b29c28d25b0" containerName="main"
Sep 16 19:44:51.698185 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.698053    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="da92812f-596d-4576-9b53-7b29c28d25b0" containerName="main"
Sep 16 19:44:51.698185 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.698065    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e33d5a4-a06c-459c-9a49-6a321141e7eb" containerName="main"
Sep 16 19:44:51.698185 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.698070    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e33d5a4-a06c-459c-9a49-6a321141e7eb" containerName="main"
Sep 16 19:44:51.698185 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.698115    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="da92812f-596d-4576-9b53-7b29c28d25b0" containerName="main"
Sep 16 19:44:51.698185 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.698122    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e33d5a4-a06c-459c-9a49-6a321141e7eb" containerName="main"
Sep 16 19:44:51.700775 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.700756    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"
Sep 16 19:44:51.715490 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.715463    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 19:44:51.715710 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.715476    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 19:44:51.715944 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.715609    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 19:44:51.716061 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.715617    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-5wcnd\""
Sep 16 19:44:51.716177 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.716092    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-fv7sl\""
Sep 16 19:44:51.718892 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.718867    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"]
Sep 16 19:44:51.772436 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.772413    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x86kl\" (UniqueName: \"kubernetes.io/projected/9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9-kube-api-access-x86kl\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6\" (UID: \"9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"
Sep 16 19:44:51.772552 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.772457    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6\" (UID: \"9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"
Sep 16 19:44:51.872872 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.872844    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6\" (UID: \"9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"
Sep 16 19:44:51.873021 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.872907    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x86kl\" (UniqueName: \"kubernetes.io/projected/9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9-kube-api-access-x86kl\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6\" (UID: \"9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"
Sep 16 19:44:51.875348 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.875323    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6\" (UID: \"9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"
Sep 16 19:44:51.881896 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:51.881865    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x86kl\" (UniqueName: \"kubernetes.io/projected/9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9-kube-api-access-x86kl\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6\" (UID: \"9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"
Sep 16 19:44:52.020113 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:52.020089    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"
Sep 16 19:44:52.287860 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:52.287825    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"]
Sep 16 19:44:52.290430 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:44:52.290399    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba9ada9_bb2d_4cf8_8be9_cd1686a418d9.slice/crio-6ecb0ebf72eaa4431c72a1e0cd2b2754e89a68dc170eb9869e9e5a2f9c561279 WatchSource:0}: Error finding container 6ecb0ebf72eaa4431c72a1e0cd2b2754e89a68dc170eb9869e9e5a2f9c561279: Status 404 returned error can't find the container with id 6ecb0ebf72eaa4431c72a1e0cd2b2754e89a68dc170eb9869e9e5a2f9c561279
Sep 16 19:44:53.004250 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:53.004211    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6" event={"ID":"9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9","Type":"ContainerStarted","Data":"64eeb93096147fae5e96e5a91c2372a6499648cca05d0d3e2c37b7ff4b76d4fd"}
Sep 16 19:44:53.004725 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:53.004271    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6" event={"ID":"9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9","Type":"ContainerStarted","Data":"6ecb0ebf72eaa4431c72a1e0cd2b2754e89a68dc170eb9869e9e5a2f9c561279"}
Sep 16 19:44:53.004725 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:53.004321    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"
Sep 16 19:44:53.046385 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:44:53.046341    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6" podStartSLOduration=2.046325316 podStartE2EDuration="2.046325316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:44:53.043814689 +0000 UTC m=+3619.428373912" watchObservedRunningTime="2026-09-16 19:44:53.046325316 +0000 UTC m=+3619.430884539"
Sep 16 19:45:24.010027 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:45:24.009995    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"
Sep 16 19:46:56.726921 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:46:56.726884    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"]
Sep 16 19:47:04.010390 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:04.010350    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6" podUID="9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:47:11.745118 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:11.745059    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6" podUID="9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9" containerName="main" containerID="cri-o://64eeb93096147fae5e96e5a91c2372a6499648cca05d0d3e2c37b7ff4b76d4fd" gracePeriod=45
Sep 16 19:47:14.010648 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:14.010613    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6" podUID="9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:47:24.010569 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:24.010530    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6" podUID="9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:47:24.010952 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:24.010628    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"
Sep 16 19:47:34.009765 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:34.009725    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6" podUID="9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:47:41.993107 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:41.993084    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"
Sep 16 19:47:42.124828 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.124799    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9-tls-certs\") pod \"9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9\" (UID: \"9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9\") "
Sep 16 19:47:42.125014 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.124858    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x86kl\" (UniqueName: \"kubernetes.io/projected/9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9-kube-api-access-x86kl\") pod \"9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9\" (UID: \"9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9\") "
Sep 16 19:47:42.126986 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.126958    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9" (UID: "9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:47:42.127083 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.127009    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9-kube-api-access-x86kl" (OuterVolumeSpecName: "kube-api-access-x86kl") pod "9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9" (UID: "9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9"). InnerVolumeSpecName "kube-api-access-x86kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:47:42.225983 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.225960    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:47:42.225983 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.225984    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x86kl\" (UniqueName: \"kubernetes.io/projected/9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9-kube-api-access-x86kl\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:47:42.577884 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.577853    2570 generic.go:356] "Generic (PLEG): container finished" podID="9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9" containerID="64eeb93096147fae5e96e5a91c2372a6499648cca05d0d3e2c37b7ff4b76d4fd" exitCode=0
Sep 16 19:47:42.578044 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.577894    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6" event={"ID":"9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9","Type":"ContainerDied","Data":"64eeb93096147fae5e96e5a91c2372a6499648cca05d0d3e2c37b7ff4b76d4fd"}
Sep 16 19:47:42.578044 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.577931    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6" event={"ID":"9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9","Type":"ContainerDied","Data":"6ecb0ebf72eaa4431c72a1e0cd2b2754e89a68dc170eb9869e9e5a2f9c561279"}
Sep 16 19:47:42.578044 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.577953    2570 scope.go:117] "RemoveContainer" containerID="64eeb93096147fae5e96e5a91c2372a6499648cca05d0d3e2c37b7ff4b76d4fd"
Sep 16 19:47:42.578044 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.577895    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"
Sep 16 19:47:42.585740 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.585720    2570 scope.go:117] "RemoveContainer" containerID="64eeb93096147fae5e96e5a91c2372a6499648cca05d0d3e2c37b7ff4b76d4fd"
Sep 16 19:47:42.586018 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:47:42.585993    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64eeb93096147fae5e96e5a91c2372a6499648cca05d0d3e2c37b7ff4b76d4fd\": container with ID starting with 64eeb93096147fae5e96e5a91c2372a6499648cca05d0d3e2c37b7ff4b76d4fd not found: ID does not exist" containerID="64eeb93096147fae5e96e5a91c2372a6499648cca05d0d3e2c37b7ff4b76d4fd"
Sep 16 19:47:42.586095 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.586026    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64eeb93096147fae5e96e5a91c2372a6499648cca05d0d3e2c37b7ff4b76d4fd"} err="failed to get container status \"64eeb93096147fae5e96e5a91c2372a6499648cca05d0d3e2c37b7ff4b76d4fd\": rpc error: code = NotFound desc = could not find container \"64eeb93096147fae5e96e5a91c2372a6499648cca05d0d3e2c37b7ff4b76d4fd\": container with ID starting with 64eeb93096147fae5e96e5a91c2372a6499648cca05d0d3e2c37b7ff4b76d4fd not found: ID does not exist"
Sep 16 19:47:42.598183 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.598159    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"]
Sep 16 19:47:42.604079 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:42.604060    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58qhdl6"]
Sep 16 19:47:44.159785 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:44.159754    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9" path="/var/lib/kubelet/pods/9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9/volumes"
Sep 16 19:47:47.093450 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.093419    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"]
Sep 16 19:47:47.093783 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.093664    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9" containerName="main"
Sep 16 19:47:47.093783 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.093679    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9" containerName="main"
Sep 16 19:47:47.093783 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.093768    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba9ada9-bb2d-4cf8-8be9-cd1686a418d9" containerName="main"
Sep 16 19:47:47.097763 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.097746    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.104789 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.104770    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 19:47:47.104882 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.104843    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 19:47:47.105086 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.105072    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:47:47.115858 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.115840    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-qvkzx\""
Sep 16 19:47:47.133010 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.132982    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"]
Sep 16 19:47:47.160122 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.160095    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.160211 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.160132    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-workload-certs\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.160211 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.160153    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-podinfo\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.160282 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.160227    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-workload-socket\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.160282 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.160255    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-token\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.160349 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.160283    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-data\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.160349 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.160315    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-credential-socket\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.160349 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.160330    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-envoy\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.160349 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.160344    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snx8f\" (UniqueName: \"kubernetes.io/projected/886de9e8-412a-4b3e-a43b-11d0d7d609cb-kube-api-access-snx8f\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.261133 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.261105    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-credential-socket\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.261251 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.261146    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-envoy\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.261251 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.261163    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snx8f\" (UniqueName: \"kubernetes.io/projected/886de9e8-412a-4b3e-a43b-11d0d7d609cb-kube-api-access-snx8f\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.261251 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.261195    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.261251 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.261223    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-workload-certs\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.261251 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.261248    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-podinfo\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.261536 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.261304    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-workload-socket\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.261536 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.261329    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-token\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.261536 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.261351    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-data\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.261666 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.261561    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-credential-socket\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.261822 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.261798    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-workload-certs\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.261895 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.261852    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-workload-socket\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.261895 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.261867    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-data\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.262017 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.261933    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.263494 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.263475    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-envoy\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.263622 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.263604    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-podinfo\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.274090 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.274067    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-token\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.274182 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.274162    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snx8f\" (UniqueName: \"kubernetes.io/projected/886de9e8-412a-4b3e-a43b-11d0d7d609cb-kube-api-access-snx8f\") pod \"router-gateway-1-openshift-default-5d66688f6-m59xb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.408006 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.407956    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:47.687371 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.687344    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"]
Sep 16 19:47:47.688740 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:47:47.688670    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886de9e8_412a_4b3e_a43b_11d0d7d609cb.slice/crio-f4d33506d444620f61949840331f1ce95a04bab7c5af4649994b8ea930df8a1b WatchSource:0}: Error finding container f4d33506d444620f61949840331f1ce95a04bab7c5af4649994b8ea930df8a1b: Status 404 returned error can't find the container with id f4d33506d444620f61949840331f1ce95a04bab7c5af4649994b8ea930df8a1b
Sep 16 19:47:47.690554 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.690524    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:47:47.690649 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.690583    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:47:47.690649 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:47.690609    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:47:48.601250 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:48.601205    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb" event={"ID":"886de9e8-412a-4b3e-a43b-11d0d7d609cb","Type":"ContainerStarted","Data":"eb64c6a9ccb4bf613ab88f568fdfa4f74ff3687d84d0d2806f02c747ff68e284"}
Sep 16 19:47:48.601661 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:48.601269    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb" event={"ID":"886de9e8-412a-4b3e-a43b-11d0d7d609cb","Type":"ContainerStarted","Data":"f4d33506d444620f61949840331f1ce95a04bab7c5af4649994b8ea930df8a1b"}
Sep 16 19:47:48.621383 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:48.621333    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb" podStartSLOduration=1.6213177380000001 podStartE2EDuration="1.621317738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:48.619995185 +0000 UTC m=+3795.004554408" watchObservedRunningTime="2026-09-16 19:47:48.621317738 +0000 UTC m=+3795.005877020"
Sep 16 19:47:49.408304 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:49.408272    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:49.409716 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:49.409665    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb" podUID="886de9e8-412a-4b3e-a43b-11d0d7d609cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 19:47:50.409074 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:50.409030    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb" podUID="886de9e8-412a-4b3e-a43b-11d0d7d609cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 19:47:51.408835 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:51.408794    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb" podUID="886de9e8-412a-4b3e-a43b-11d0d7d609cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 19:47:52.409341 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:52.409298    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb" podUID="886de9e8-412a-4b3e-a43b-11d0d7d609cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 19:47:52.431312 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:52.431276    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"]
Sep 16 19:47:52.434078 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:52.434060    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"
Sep 16 19:47:52.437307 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:52.437286    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-rpc69\""
Sep 16 19:47:52.437449 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:52.437432    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 19:47:52.437503 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:52.437462    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-bcwmw\""
Sep 16 19:47:52.451641 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:52.450993    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"]
Sep 16 19:47:52.502811 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:52.502785    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56918467-79a5-4cfa-bab9-3c43a64a40ba-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4\" (UID: \"56918467-79a5-4cfa-bab9-3c43a64a40ba\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"
Sep 16 19:47:52.502944 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:52.502844    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbgjs\" (UniqueName: \"kubernetes.io/projected/56918467-79a5-4cfa-bab9-3c43a64a40ba-kube-api-access-zbgjs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4\" (UID: \"56918467-79a5-4cfa-bab9-3c43a64a40ba\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"
Sep 16 19:47:52.603820 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:52.603789    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbgjs\" (UniqueName: \"kubernetes.io/projected/56918467-79a5-4cfa-bab9-3c43a64a40ba-kube-api-access-zbgjs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4\" (UID: \"56918467-79a5-4cfa-bab9-3c43a64a40ba\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"
Sep 16 19:47:52.603941 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:52.603834    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56918467-79a5-4cfa-bab9-3c43a64a40ba-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4\" (UID: \"56918467-79a5-4cfa-bab9-3c43a64a40ba\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"
Sep 16 19:47:52.606352 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:52.606326    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56918467-79a5-4cfa-bab9-3c43a64a40ba-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4\" (UID: \"56918467-79a5-4cfa-bab9-3c43a64a40ba\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"
Sep 16 19:47:52.613800 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:52.613775    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbgjs\" (UniqueName: \"kubernetes.io/projected/56918467-79a5-4cfa-bab9-3c43a64a40ba-kube-api-access-zbgjs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4\" (UID: \"56918467-79a5-4cfa-bab9-3c43a64a40ba\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"
Sep 16 19:47:52.762114 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:52.762022    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"
Sep 16 19:47:53.032766 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:53.032735    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"]
Sep 16 19:47:53.035577 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:47:53.035549    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56918467_79a5_4cfa_bab9_3c43a64a40ba.slice/crio-be07e4d2ea7f274e4da47a915830a3c5331f1038737d73285f05205f4ea163f0 WatchSource:0}: Error finding container be07e4d2ea7f274e4da47a915830a3c5331f1038737d73285f05205f4ea163f0: Status 404 returned error can't find the container with id be07e4d2ea7f274e4da47a915830a3c5331f1038737d73285f05205f4ea163f0
Sep 16 19:47:53.409311 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:53.409214    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb" podUID="886de9e8-412a-4b3e-a43b-11d0d7d609cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 16 19:47:53.621884 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:53.621852    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4" event={"ID":"56918467-79a5-4cfa-bab9-3c43a64a40ba","Type":"ContainerStarted","Data":"788bf3dbfe754d385b787398a23e0c45b05c0e337a61e8d74999c1bd107002ec"}
Sep 16 19:47:53.622279 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:53.621903    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4" event={"ID":"56918467-79a5-4cfa-bab9-3c43a64a40ba","Type":"ContainerStarted","Data":"be07e4d2ea7f274e4da47a915830a3c5331f1038737d73285f05205f4ea163f0"}
Sep 16 19:47:53.622279 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:53.621955    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"
Sep 16 19:47:53.642326 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:53.642271    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4" podStartSLOduration=1.6422539569999999 podStartE2EDuration="1.642253957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:53.640976345 +0000 UTC m=+3800.025535592" watchObservedRunningTime="2026-09-16 19:47:53.642253957 +0000 UTC m=+3800.026813179"
Sep 16 19:47:54.413037 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:54.413005    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:54.413294 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:54.413274    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:47:54.414006 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:47:54.413983    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:48:24.627924 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:48:24.627891    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"
Sep 16 19:49:57.569258 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:49:57.569214    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"]
Sep 16 19:50:04.627379 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:04.627340    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4" podUID="56918467-79a5-4cfa-bab9-3c43a64a40ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:50:12.587677 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:12.587636    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4" podUID="56918467-79a5-4cfa-bab9-3c43a64a40ba" containerName="main" containerID="cri-o://788bf3dbfe754d385b787398a23e0c45b05c0e337a61e8d74999c1bd107002ec" gracePeriod=45
Sep 16 19:50:14.627903 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:14.627869    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4" podUID="56918467-79a5-4cfa-bab9-3c43a64a40ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:50:24.627598 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:24.627558    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4" podUID="56918467-79a5-4cfa-bab9-3c43a64a40ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:50:24.628060 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:24.627660    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"
Sep 16 19:50:34.627628 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:34.627589    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4" podUID="56918467-79a5-4cfa-bab9-3c43a64a40ba" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:50:42.825622 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:42.825597    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"
Sep 16 19:50:42.866470 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:42.866411    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbgjs\" (UniqueName: \"kubernetes.io/projected/56918467-79a5-4cfa-bab9-3c43a64a40ba-kube-api-access-zbgjs\") pod \"56918467-79a5-4cfa-bab9-3c43a64a40ba\" (UID: \"56918467-79a5-4cfa-bab9-3c43a64a40ba\") "
Sep 16 19:50:42.866578 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:42.866473    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56918467-79a5-4cfa-bab9-3c43a64a40ba-tls-certs\") pod \"56918467-79a5-4cfa-bab9-3c43a64a40ba\" (UID: \"56918467-79a5-4cfa-bab9-3c43a64a40ba\") "
Sep 16 19:50:42.868571 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:42.868540    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56918467-79a5-4cfa-bab9-3c43a64a40ba-kube-api-access-zbgjs" (OuterVolumeSpecName: "kube-api-access-zbgjs") pod "56918467-79a5-4cfa-bab9-3c43a64a40ba" (UID: "56918467-79a5-4cfa-bab9-3c43a64a40ba"). InnerVolumeSpecName "kube-api-access-zbgjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:50:42.868676 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:42.868611    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56918467-79a5-4cfa-bab9-3c43a64a40ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56918467-79a5-4cfa-bab9-3c43a64a40ba" (UID: "56918467-79a5-4cfa-bab9-3c43a64a40ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:50:42.967674 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:42.967648    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56918467-79a5-4cfa-bab9-3c43a64a40ba-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:50:42.967674 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:42.967670    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbgjs\" (UniqueName: \"kubernetes.io/projected/56918467-79a5-4cfa-bab9-3c43a64a40ba-kube-api-access-zbgjs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:50:43.229735 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:43.229636    2570 generic.go:356] "Generic (PLEG): container finished" podID="56918467-79a5-4cfa-bab9-3c43a64a40ba" containerID="788bf3dbfe754d385b787398a23e0c45b05c0e337a61e8d74999c1bd107002ec" exitCode=0
Sep 16 19:50:43.229735 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:43.229677    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"
Sep 16 19:50:43.229953 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:43.229724    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4" event={"ID":"56918467-79a5-4cfa-bab9-3c43a64a40ba","Type":"ContainerDied","Data":"788bf3dbfe754d385b787398a23e0c45b05c0e337a61e8d74999c1bd107002ec"}
Sep 16 19:50:43.229953 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:43.229783    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4" event={"ID":"56918467-79a5-4cfa-bab9-3c43a64a40ba","Type":"ContainerDied","Data":"be07e4d2ea7f274e4da47a915830a3c5331f1038737d73285f05205f4ea163f0"}
Sep 16 19:50:43.229953 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:43.229812    2570 scope.go:117] "RemoveContainer" containerID="788bf3dbfe754d385b787398a23e0c45b05c0e337a61e8d74999c1bd107002ec"
Sep 16 19:50:43.237959 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:43.237942    2570 scope.go:117] "RemoveContainer" containerID="788bf3dbfe754d385b787398a23e0c45b05c0e337a61e8d74999c1bd107002ec"
Sep 16 19:50:43.238197 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:50:43.238179    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788bf3dbfe754d385b787398a23e0c45b05c0e337a61e8d74999c1bd107002ec\": container with ID starting with 788bf3dbfe754d385b787398a23e0c45b05c0e337a61e8d74999c1bd107002ec not found: ID does not exist" containerID="788bf3dbfe754d385b787398a23e0c45b05c0e337a61e8d74999c1bd107002ec"
Sep 16 19:50:43.238267 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:43.238203    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788bf3dbfe754d385b787398a23e0c45b05c0e337a61e8d74999c1bd107002ec"} err="failed to get container status \"788bf3dbfe754d385b787398a23e0c45b05c0e337a61e8d74999c1bd107002ec\": rpc error: code = NotFound desc = could not find container \"788bf3dbfe754d385b787398a23e0c45b05c0e337a61e8d74999c1bd107002ec\": container with ID starting with 788bf3dbfe754d385b787398a23e0c45b05c0e337a61e8d74999c1bd107002ec not found: ID does not exist"
Sep 16 19:50:43.252515 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:43.252492    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"]
Sep 16 19:50:43.259042 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:43.259022    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-vwkt4"]
Sep 16 19:50:44.160300 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:44.160265    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56918467-79a5-4cfa-bab9-3c43a64a40ba" path="/var/lib/kubelet/pods/56918467-79a5-4cfa-bab9-3c43a64a40ba/volumes"
Sep 16 19:50:52.403530 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:52.403497    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"]
Sep 16 19:50:52.404023 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:52.403707    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb" podUID="886de9e8-412a-4b3e-a43b-11d0d7d609cb" containerName="istio-proxy" containerID="cri-o://eb64c6a9ccb4bf613ab88f568fdfa4f74ff3687d84d0d2806f02c747ff68e284" gracePeriod=30
Sep 16 19:50:54.416237 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:54.416196    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb" podUID="886de9e8-412a-4b3e-a43b-11d0d7d609cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:50:57.643762 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.643738    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:50:57.771902 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.771872    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-workload-certs\") pod \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") "
Sep 16 19:50:57.772075 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.771907    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-credential-socket\") pod \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") "
Sep 16 19:50:57.772075 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.771932    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-workload-socket\") pod \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") "
Sep 16 19:50:57.772075 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.771949    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-podinfo\") pod \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") "
Sep 16 19:50:57.772075 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.771989    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snx8f\" (UniqueName: \"kubernetes.io/projected/886de9e8-412a-4b3e-a43b-11d0d7d609cb-kube-api-access-snx8f\") pod \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") "
Sep 16 19:50:57.772075 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.772012    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istiod-ca-cert\") pod \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") "
Sep 16 19:50:57.772075 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.772030    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-token\") pod \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") "
Sep 16 19:50:57.772075 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.772052    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-data\") pod \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") "
Sep 16 19:50:57.772439 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.772093    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-envoy\") pod \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\" (UID: \"886de9e8-412a-4b3e-a43b-11d0d7d609cb\") "
Sep 16 19:50:57.772439 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.772166    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "886de9e8-412a-4b3e-a43b-11d0d7d609cb" (UID: "886de9e8-412a-4b3e-a43b-11d0d7d609cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:57.772439 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.772214    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "886de9e8-412a-4b3e-a43b-11d0d7d609cb" (UID: "886de9e8-412a-4b3e-a43b-11d0d7d609cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:57.772439 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.772186    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "886de9e8-412a-4b3e-a43b-11d0d7d609cb" (UID: "886de9e8-412a-4b3e-a43b-11d0d7d609cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:57.772439 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.772312    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-workload-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:50:57.772439 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.772353    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-data" (OuterVolumeSpecName: "istio-data") pod "886de9e8-412a-4b3e-a43b-11d0d7d609cb" (UID: "886de9e8-412a-4b3e-a43b-11d0d7d609cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:57.772439 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.772361    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-credential-socket\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:50:57.772439 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.772377    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-workload-socket\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:50:57.772877 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.772605    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "886de9e8-412a-4b3e-a43b-11d0d7d609cb" (UID: "886de9e8-412a-4b3e-a43b-11d0d7d609cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:50:57.774315 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.774284    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-token" (OuterVolumeSpecName: "istio-token") pod "886de9e8-412a-4b3e-a43b-11d0d7d609cb" (UID: "886de9e8-412a-4b3e-a43b-11d0d7d609cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:50:57.774431 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.774364    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "886de9e8-412a-4b3e-a43b-11d0d7d609cb" (UID: "886de9e8-412a-4b3e-a43b-11d0d7d609cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:50:57.774431 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.774368    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886de9e8-412a-4b3e-a43b-11d0d7d609cb-kube-api-access-snx8f" (OuterVolumeSpecName: "kube-api-access-snx8f") pod "886de9e8-412a-4b3e-a43b-11d0d7d609cb" (UID: "886de9e8-412a-4b3e-a43b-11d0d7d609cb"). InnerVolumeSpecName "kube-api-access-snx8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:50:57.774431 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.774402    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "886de9e8-412a-4b3e-a43b-11d0d7d609cb" (UID: "886de9e8-412a-4b3e-a43b-11d0d7d609cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:50:57.872998 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.872967    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-envoy\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:50:57.872998 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.872996    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-podinfo\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:50:57.872998 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.873005    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snx8f\" (UniqueName: \"kubernetes.io/projected/886de9e8-412a-4b3e-a43b-11d0d7d609cb-kube-api-access-snx8f\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:50:57.873174 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.873014    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istiod-ca-cert\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:50:57.873174 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.873022    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-token\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:50:57.873174 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:57.873030    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/886de9e8-412a-4b3e-a43b-11d0d7d609cb-istio-data\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:50:58.283543 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:58.283516    2570 generic.go:356] "Generic (PLEG): container finished" podID="886de9e8-412a-4b3e-a43b-11d0d7d609cb" containerID="eb64c6a9ccb4bf613ab88f568fdfa4f74ff3687d84d0d2806f02c747ff68e284" exitCode=0
Sep 16 19:50:58.283670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:58.283556    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb" event={"ID":"886de9e8-412a-4b3e-a43b-11d0d7d609cb","Type":"ContainerDied","Data":"eb64c6a9ccb4bf613ab88f568fdfa4f74ff3687d84d0d2806f02c747ff68e284"}
Sep 16 19:50:58.283670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:58.283568    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"
Sep 16 19:50:58.283670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:58.283588    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb" event={"ID":"886de9e8-412a-4b3e-a43b-11d0d7d609cb","Type":"ContainerDied","Data":"f4d33506d444620f61949840331f1ce95a04bab7c5af4649994b8ea930df8a1b"}
Sep 16 19:50:58.283670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:58.283609    2570 scope.go:117] "RemoveContainer" containerID="eb64c6a9ccb4bf613ab88f568fdfa4f74ff3687d84d0d2806f02c747ff68e284"
Sep 16 19:50:58.291709 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:58.291676    2570 scope.go:117] "RemoveContainer" containerID="eb64c6a9ccb4bf613ab88f568fdfa4f74ff3687d84d0d2806f02c747ff68e284"
Sep 16 19:50:58.291994 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:50:58.291972    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb64c6a9ccb4bf613ab88f568fdfa4f74ff3687d84d0d2806f02c747ff68e284\": container with ID starting with eb64c6a9ccb4bf613ab88f568fdfa4f74ff3687d84d0d2806f02c747ff68e284 not found: ID does not exist" containerID="eb64c6a9ccb4bf613ab88f568fdfa4f74ff3687d84d0d2806f02c747ff68e284"
Sep 16 19:50:58.292050 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:58.292001    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb64c6a9ccb4bf613ab88f568fdfa4f74ff3687d84d0d2806f02c747ff68e284"} err="failed to get container status \"eb64c6a9ccb4bf613ab88f568fdfa4f74ff3687d84d0d2806f02c747ff68e284\": rpc error: code = NotFound desc = could not find container \"eb64c6a9ccb4bf613ab88f568fdfa4f74ff3687d84d0d2806f02c747ff68e284\": container with ID starting with eb64c6a9ccb4bf613ab88f568fdfa4f74ff3687d84d0d2806f02c747ff68e284 not found: ID does not exist"
Sep 16 19:50:58.301556 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:58.301534    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"]
Sep 16 19:50:58.304015 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:50:58.303994    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-5d66688f6-m59xb"]
Sep 16 19:51:00.160659 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:00.160627    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="886de9e8-412a-4b3e-a43b-11d0d7d609cb" path="/var/lib/kubelet/pods/886de9e8-412a-4b3e-a43b-11d0d7d609cb/volumes"
Sep 16 19:51:10.146925 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.146888    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"]
Sep 16 19:51:10.147336 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.147129    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="886de9e8-412a-4b3e-a43b-11d0d7d609cb" containerName="istio-proxy"
Sep 16 19:51:10.147336 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.147139    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="886de9e8-412a-4b3e-a43b-11d0d7d609cb" containerName="istio-proxy"
Sep 16 19:51:10.147336 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.147148    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56918467-79a5-4cfa-bab9-3c43a64a40ba" containerName="main"
Sep 16 19:51:10.147336 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.147154    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="56918467-79a5-4cfa-bab9-3c43a64a40ba" containerName="main"
Sep 16 19:51:10.147336 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.147196    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="886de9e8-412a-4b3e-a43b-11d0d7d609cb" containerName="istio-proxy"
Sep 16 19:51:10.147336 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.147204    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="56918467-79a5-4cfa-bab9-3c43a64a40ba" containerName="main"
Sep 16 19:51:10.150166 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.150149    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.152558 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.152516    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:51:10.152716 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.152562    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 19:51:10.152716 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.152667    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 19:51:10.152827 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.152766    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-vzp6c\""
Sep 16 19:51:10.152959 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.152942    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-mkkwr\""
Sep 16 19:51:10.153019 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.153002    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 19:51:10.153361 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.153345    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 19:51:10.166267 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.166246    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"]
Sep 16 19:51:10.252860 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.252833    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.253024 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.252886    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.253024 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.252922    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.253024 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.252957    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.253024 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.252985    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.253249 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.253113    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx7bg\" (UniqueName: \"kubernetes.io/projected/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-kube-api-access-cx7bg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.253249 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.253146    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.253249 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.253235    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.354080 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.354054    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.354236 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.354113    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.354236 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.354145    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.354236 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.354189    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.354236 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.354218    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.354446 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.354244    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.354446 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.354272    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.354446 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.354314    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx7bg\" (UniqueName: \"kubernetes.io/projected/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-kube-api-access-cx7bg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.354596 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.354479    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.354596 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.354519    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.354789 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.354766    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.354910 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.354889    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.355023 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.355009    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.356285 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.356260    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.356580 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.356560    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.363052 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.363032    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx7bg\" (UniqueName: \"kubernetes.io/projected/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-kube-api-access-cx7bg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.459580 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.459514    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:10.742590 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:10.742562    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"]
Sep 16 19:51:10.743673 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:51:10.743649    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bcf1c0_1b79_4cca_97bf_2e4e0cca0046.slice/crio-dc81b15cc7b5e73417fe1df9660d3b5ac642c46b067c8aac468b8f1cb9b34599 WatchSource:0}: Error finding container dc81b15cc7b5e73417fe1df9660d3b5ac642c46b067c8aac468b8f1cb9b34599: Status 404 returned error can't find the container with id dc81b15cc7b5e73417fe1df9660d3b5ac642c46b067c8aac468b8f1cb9b34599
Sep 16 19:51:11.329947 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:11.329914    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" event={"ID":"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046","Type":"ContainerStarted","Data":"d2f2badfd8ef76321873b9334292bf22aba05a4841d8393bb657cefd37a0d8d4"}
Sep 16 19:51:11.330370 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:11.329967    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" event={"ID":"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046","Type":"ContainerStarted","Data":"dc81b15cc7b5e73417fe1df9660d3b5ac642c46b067c8aac468b8f1cb9b34599"}
Sep 16 19:51:11.330370 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:11.329997    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:12.338460 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:12.338413    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" event={"ID":"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046","Type":"ContainerStarted","Data":"b7a62aa6382d0c0147312f621540abbdf6f060045b5aa2acf933028d4609b095"}
Sep 16 19:51:14.352581 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:14.352539    2570 generic.go:356] "Generic (PLEG): container finished" podID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerID="b7a62aa6382d0c0147312f621540abbdf6f060045b5aa2acf933028d4609b095" exitCode=0
Sep 16 19:51:14.353084 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:14.352658    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" event={"ID":"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046","Type":"ContainerDied","Data":"b7a62aa6382d0c0147312f621540abbdf6f060045b5aa2acf933028d4609b095"}
Sep 16 19:51:15.360711 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:15.360649    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" event={"ID":"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046","Type":"ContainerStarted","Data":"6e04b21943ca3a55a188ef393a1559a9fd56d095ddccab9165337fc559d271cb"}
Sep 16 19:51:15.388376 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:15.388321    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" podStartSLOduration=2.899614923 podStartE2EDuration="5.38830081s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.488685887s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:51:15.385894207 +0000 UTC m=+4001.770453445" watchObservedRunningTime="2026-09-16 19:51:15.38830081 +0000 UTC m=+4001.772860036"
Sep 16 19:51:20.459883 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:20.459841    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:20.460295 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:20.459917    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:20.461167 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:20.461115    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Sep 16 19:51:20.473822 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:20.473792    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:51:30.460569 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:30.460526    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Sep 16 19:51:40.460582 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:40.460531    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Sep 16 19:51:50.460026 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:51:50.459988    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Sep 16 19:52:00.460248 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:52:00.460133    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Sep 16 19:52:10.460320 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:52:10.460272    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Sep 16 19:52:20.460124 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:52:20.460084    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Sep 16 19:52:30.460894 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:52:30.460846    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Sep 16 19:52:40.460387 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:52:40.460327    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Sep 16 19:52:50.460337 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:52:50.460292    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Sep 16 19:53:00.460842 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:00.460794    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8001/health\": dial tcp 10.132.0.49:8001: connect: connection refused"
Sep 16 19:53:10.469950 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:10.469919    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:53:10.485920 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:10.485899    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:53:20.163214 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:20.163176    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"]
Sep 16 19:53:35.185302 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:35.185242    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main" containerID="cri-o://6e04b21943ca3a55a188ef393a1559a9fd56d095ddccab9165337fc559d271cb" gracePeriod=45
Sep 16 19:53:35.701140 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:35.701085    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="llm-d-routing-sidecar" containerID="cri-o://d2f2badfd8ef76321873b9334292bf22aba05a4841d8393bb657cefd37a0d8d4" gracePeriod=45
Sep 16 19:53:35.897987 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:35.897917    2570 generic.go:356] "Generic (PLEG): container finished" podID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerID="6e04b21943ca3a55a188ef393a1559a9fd56d095ddccab9165337fc559d271cb" exitCode=0
Sep 16 19:53:35.897987 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:35.897940    2570 generic.go:356] "Generic (PLEG): container finished" podID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerID="d2f2badfd8ef76321873b9334292bf22aba05a4841d8393bb657cefd37a0d8d4" exitCode=0
Sep 16 19:53:35.898143 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:35.897992    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" event={"ID":"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046","Type":"ContainerDied","Data":"6e04b21943ca3a55a188ef393a1559a9fd56d095ddccab9165337fc559d271cb"}
Sep 16 19:53:35.898143 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:35.898100    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" event={"ID":"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046","Type":"ContainerDied","Data":"d2f2badfd8ef76321873b9334292bf22aba05a4841d8393bb657cefd37a0d8d4"}
Sep 16 19:53:35.930547 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:35.930522    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:53:36.029983 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.029958    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-tmp-dir\") pod \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") "
Sep 16 19:53:36.030121 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.029998    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-dshm\") pod \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") "
Sep 16 19:53:36.030121 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.030042    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-tls-certs\") pod \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") "
Sep 16 19:53:36.030121 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.030056    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-home\") pod \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") "
Sep 16 19:53:36.030121 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.030071    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-cabundle-cert\") pod \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") "
Sep 16 19:53:36.030121 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.030097    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-kserve-provision-location\") pod \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") "
Sep 16 19:53:36.030398 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.030125    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-model-cache\") pod \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") "
Sep 16 19:53:36.030398 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.030158    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cx7bg\" (UniqueName: \"kubernetes.io/projected/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-kube-api-access-cx7bg\") pod \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\" (UID: \"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046\") "
Sep 16 19:53:36.030537 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.030507    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" (UID: "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:53:36.030930 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.030901    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-model-cache" (OuterVolumeSpecName: "model-cache") pod "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" (UID: "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:36.031092 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.030990    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-home" (OuterVolumeSpecName: "home") pod "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" (UID: "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:36.032477 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.032446    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-dshm" (OuterVolumeSpecName: "dshm") pod "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" (UID: "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:36.032575 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.032510    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-kube-api-access-cx7bg" (OuterVolumeSpecName: "kube-api-access-cx7bg") pod "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" (UID: "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046"). InnerVolumeSpecName "kube-api-access-cx7bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:53:36.032575 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.032532    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" (UID: "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:53:36.042383 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.042359    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" (UID: "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:36.051827 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.051798    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" (UID: "52bcf1c0-1b79-4cca-97bf-2e4e0cca0046"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:36.131417 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.131393    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.131417 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.131415    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.131567 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.131427    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-cabundle-cert\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.131567 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.131440    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-kserve-provision-location\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.131567 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.131453    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.131567 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.131465    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx7bg\" (UniqueName: \"kubernetes.io/projected/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-kube-api-access-cx7bg\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.131567 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.131503    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.131567 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.131518    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:53:36.903567 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.903531    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2" event={"ID":"52bcf1c0-1b79-4cca-97bf-2e4e0cca0046","Type":"ContainerDied","Data":"dc81b15cc7b5e73417fe1df9660d3b5ac642c46b067c8aac468b8f1cb9b34599"}
Sep 16 19:53:36.903567 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.903562    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"
Sep 16 19:53:36.904119 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.903595    2570 scope.go:117] "RemoveContainer" containerID="6e04b21943ca3a55a188ef393a1559a9fd56d095ddccab9165337fc559d271cb"
Sep 16 19:53:36.911484 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.911468    2570 scope.go:117] "RemoveContainer" containerID="b7a62aa6382d0c0147312f621540abbdf6f060045b5aa2acf933028d4609b095"
Sep 16 19:53:36.918419 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.918399    2570 scope.go:117] "RemoveContainer" containerID="d2f2badfd8ef76321873b9334292bf22aba05a4841d8393bb657cefd37a0d8d4"
Sep 16 19:53:36.922411 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.922391    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"]
Sep 16 19:53:36.928246 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:36.928226    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67966dc77b559b2"]
Sep 16 19:53:38.159461 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:53:38.159428    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" path="/var/lib/kubelet/pods/52bcf1c0-1b79-4cca-97bf-2e4e0cca0046/volumes"
Sep 16 19:54:12.358970 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.358933    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"]
Sep 16 19:54:12.359433 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.359187    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="llm-d-routing-sidecar"
Sep 16 19:54:12.359433 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.359200    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="llm-d-routing-sidecar"
Sep 16 19:54:12.359433 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.359213    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="storage-initializer"
Sep 16 19:54:12.359433 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.359219    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="storage-initializer"
Sep 16 19:54:12.359433 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.359225    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main"
Sep 16 19:54:12.359433 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.359230    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main"
Sep 16 19:54:12.359433 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.359280    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="llm-d-routing-sidecar"
Sep 16 19:54:12.359433 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.359287    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="52bcf1c0-1b79-4cca-97bf-2e4e0cca0046" containerName="main"
Sep 16 19:54:12.361921 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.361904    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"
Sep 16 19:54:12.364454 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.364429    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 16 19:54:12.365164 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.365143    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 16 19:54:12.365334 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.365313    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-bhdh2\""
Sep 16 19:54:12.365489 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.365465    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 19:54:12.365590 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.365471    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-smjhg\""
Sep 16 19:54:12.371970 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.371951    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"]
Sep 16 19:54:12.474250 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.474217    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af11391-1a6f-44c5-afaf-f3270f96f34e-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f\" (UID: \"7af11391-1a6f-44c5-afaf-f3270f96f34e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"
Sep 16 19:54:12.474434 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.474266    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbjw6\" (UniqueName: \"kubernetes.io/projected/7af11391-1a6f-44c5-afaf-f3270f96f34e-kube-api-access-kbjw6\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f\" (UID: \"7af11391-1a6f-44c5-afaf-f3270f96f34e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"
Sep 16 19:54:12.575522 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.575489    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbjw6\" (UniqueName: \"kubernetes.io/projected/7af11391-1a6f-44c5-afaf-f3270f96f34e-kube-api-access-kbjw6\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f\" (UID: \"7af11391-1a6f-44c5-afaf-f3270f96f34e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"
Sep 16 19:54:12.575682 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.575550    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af11391-1a6f-44c5-afaf-f3270f96f34e-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f\" (UID: \"7af11391-1a6f-44c5-afaf-f3270f96f34e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"
Sep 16 19:54:12.577965 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.577940    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af11391-1a6f-44c5-afaf-f3270f96f34e-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f\" (UID: \"7af11391-1a6f-44c5-afaf-f3270f96f34e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"
Sep 16 19:54:12.584663 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.584642    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbjw6\" (UniqueName: \"kubernetes.io/projected/7af11391-1a6f-44c5-afaf-f3270f96f34e-kube-api-access-kbjw6\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f\" (UID: \"7af11391-1a6f-44c5-afaf-f3270f96f34e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"
Sep 16 19:54:12.672169 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.672111    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"
Sep 16 19:54:12.948576 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:12.948553    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"]
Sep 16 19:54:12.950848 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:54:12.950817    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af11391_1a6f_44c5_afaf_f3270f96f34e.slice/crio-b6da369994493046b7c5d4c9dcf61fed96665aed88224ba0db5e4dac87724434 WatchSource:0}: Error finding container b6da369994493046b7c5d4c9dcf61fed96665aed88224ba0db5e4dac87724434: Status 404 returned error can't find the container with id b6da369994493046b7c5d4c9dcf61fed96665aed88224ba0db5e4dac87724434
Sep 16 19:54:13.030581 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:13.030548    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f" event={"ID":"7af11391-1a6f-44c5-afaf-f3270f96f34e","Type":"ContainerStarted","Data":"42dbafab60a086e445791e593a3718e862bd08c064ad0440a8442b639f54ba05"}
Sep 16 19:54:13.030726 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:13.030709    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"
Sep 16 19:54:13.030782 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:13.030745    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f" event={"ID":"7af11391-1a6f-44c5-afaf-f3270f96f34e","Type":"ContainerStarted","Data":"b6da369994493046b7c5d4c9dcf61fed96665aed88224ba0db5e4dac87724434"}
Sep 16 19:54:13.049813 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:13.049764    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f" podStartSLOduration=1.04974916 podStartE2EDuration="1.04974916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:54:13.048049788 +0000 UTC m=+4179.432609053" watchObservedRunningTime="2026-09-16 19:54:13.04974916 +0000 UTC m=+4179.434308382"
Sep 16 19:54:26.739224 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.739172    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"]
Sep 16 19:54:26.742243 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.742223    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.745397 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.745374    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:54:26.745718 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.745674    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-xxrsk\""
Sep 16 19:54:26.745718 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.745710    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 19:54:26.745890 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.745757    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 19:54:26.745890 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.745710    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 19:54:26.745890 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.745833    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 19:54:26.762778 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.762754    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"]
Sep 16 19:54:26.869381 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.869361    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.869500 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.869386    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.869500 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.869409    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.869500 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.869430    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.869663 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.869544    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.869663 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.869585    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.869663 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.869644    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.869806 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.869678    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b749v\" (UniqueName: \"kubernetes.io/projected/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-kube-api-access-b749v\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.970659 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.970628    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.970659 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.970662    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.970895 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.970713    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b749v\" (UniqueName: \"kubernetes.io/projected/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-kube-api-access-b749v\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.970895 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.970732    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.970895 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.970746    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.970895 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.970781    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.970895 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.970808    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.970895 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.970852    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.971223 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.971080    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.971223 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.971093    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.971339 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.971266    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.971398 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.971366    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.971595 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.971575    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.973299 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.973279    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.973865 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.973843    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:26.992373 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:26.992326    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b749v\" (UniqueName: \"kubernetes.io/projected/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-kube-api-access-b749v\") pod \"custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:27.052909 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:27.052887    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:27.328388 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:27.328364    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"]
Sep 16 19:54:27.330814 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:54:27.330789    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37e5925_ecd0_48a8_8beb_db0d2f5ab62a.slice/crio-6b6184dcf0718ae1699031d210cba54290821a9c26427117cdf326a41a050884 WatchSource:0}: Error finding container 6b6184dcf0718ae1699031d210cba54290821a9c26427117cdf326a41a050884: Status 404 returned error can't find the container with id 6b6184dcf0718ae1699031d210cba54290821a9c26427117cdf326a41a050884
Sep 16 19:54:28.086581 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:28.086541    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" event={"ID":"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a","Type":"ContainerStarted","Data":"8b162c68a007b018b95afe620d1a7fda95e8eeeb34cb55307fdbfbbee7f145a2"}
Sep 16 19:54:28.087017 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:28.086600    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" event={"ID":"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a","Type":"ContainerStarted","Data":"6b6184dcf0718ae1699031d210cba54290821a9c26427117cdf326a41a050884"}
Sep 16 19:54:30.096358 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:30.096282    2570 generic.go:356] "Generic (PLEG): container finished" podID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerID="8b162c68a007b018b95afe620d1a7fda95e8eeeb34cb55307fdbfbbee7f145a2" exitCode=0
Sep 16 19:54:30.096358 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:30.096323    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" event={"ID":"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a","Type":"ContainerDied","Data":"8b162c68a007b018b95afe620d1a7fda95e8eeeb34cb55307fdbfbbee7f145a2"}
Sep 16 19:54:31.104102 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:31.104061    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" event={"ID":"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a","Type":"ContainerStarted","Data":"b002fac7a0b8257754af315e1c00a446e1bedd6f8fa3355e65d0a0990baea7a4"}
Sep 16 19:54:37.054044 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:37.053999    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:37.054478 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:37.054166    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:54:37.055842 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:37.055816    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:54:44.036023 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:44.035993    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"
Sep 16 19:54:44.054965 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:44.054909    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" podStartSLOduration=15.94260387 podStartE2EDuration="18.054889055s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.112285185s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:54:31.151919958 +0000 UTC m=+4197.536479192" watchObservedRunningTime="2026-09-16 19:54:44.054889055 +0000 UTC m=+4210.439448280"
Sep 16 19:54:47.054136 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:47.054098    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:54:57.053625 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:54:57.053531    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:55:07.053565 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:55:07.053521    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:55:17.054112 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:55:17.054066    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:55:27.054151 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:55:27.054103    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:55:37.053644 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:55:37.053591    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:55:47.054007 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:55:47.053955    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:55:57.054115 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:55:57.054064    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:56:07.053697 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:07.053650    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 16 19:56:16.997469 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:16.997437    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"]
Sep 16 19:56:17.066210 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:17.066161    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:56:17.074006 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:17.073973    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:56:24.035705 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:24.035652    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f" podUID="7af11391-1a6f-44c5-afaf-f3270f96f34e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:32.015938 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:32.015893    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f" podUID="7af11391-1a6f-44c5-afaf-f3270f96f34e" containerName="main" containerID="cri-o://42dbafab60a086e445791e593a3718e862bd08c064ad0440a8442b639f54ba05" gracePeriod=45
Sep 16 19:56:34.036439 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:34.036399    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f" podUID="7af11391-1a6f-44c5-afaf-f3270f96f34e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:36.335614 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:36.335582    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"]
Sep 16 19:56:44.035608 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:44.035572    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f" podUID="7af11391-1a6f-44c5-afaf-f3270f96f34e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:44.036073 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:44.035673    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"
Sep 16 19:56:51.357561 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:51.357492    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="main" containerID="cri-o://b002fac7a0b8257754af315e1c00a446e1bedd6f8fa3355e65d0a0990baea7a4" gracePeriod=45
Sep 16 19:56:52.098770 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.098744    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:56:52.189067 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.189006    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-kserve-provision-location\") pod \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") "
Sep 16 19:56:52.189067 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.189040    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-cabundle-cert\") pod \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") "
Sep 16 19:56:52.189263 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.189076    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-home\") pod \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") "
Sep 16 19:56:52.189263 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.189097    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-tmp-dir\") pod \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") "
Sep 16 19:56:52.189263 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.189123    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-tls-certs\") pod \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") "
Sep 16 19:56:52.189263 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.189144    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-model-cache\") pod \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") "
Sep 16 19:56:52.189263 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.189162    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b749v\" (UniqueName: \"kubernetes.io/projected/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-kube-api-access-b749v\") pod \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") "
Sep 16 19:56:52.189263 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.189186    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-dshm\") pod \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\" (UID: \"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a\") "
Sep 16 19:56:52.189570 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.189518    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" (UID: "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:52.189570 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.189524    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" (UID: "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:56:52.190043 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.190007    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-home" (OuterVolumeSpecName: "home") pod "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" (UID: "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:52.191525 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.191491    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-kube-api-access-b749v" (OuterVolumeSpecName: "kube-api-access-b749v") pod "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" (UID: "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a"). InnerVolumeSpecName "kube-api-access-b749v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:56:52.191631 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.191581    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" (UID: "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:56:52.191763 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.191701    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-dshm" (OuterVolumeSpecName: "dshm") pod "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" (UID: "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:52.202447 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.202423    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" (UID: "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:52.211088 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.211069    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" (UID: "d37e5925-ecd0-48a8-8beb-db0d2f5ab62a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:56:52.290091 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.290070    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-cabundle-cert\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.290091 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.290089    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.290200 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.290114    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.290200 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.290123    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.290200 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.290131    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.290200 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.290138    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b749v\" (UniqueName: \"kubernetes.io/projected/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-kube-api-access-b749v\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.290200 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.290147    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.290200 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.290155    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a-kserve-provision-location\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:56:52.632418 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.632384    2570 generic.go:356] "Generic (PLEG): container finished" podID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerID="b002fac7a0b8257754af315e1c00a446e1bedd6f8fa3355e65d0a0990baea7a4" exitCode=0
Sep 16 19:56:52.632784 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.632430    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"
Sep 16 19:56:52.632784 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.632474    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" event={"ID":"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a","Type":"ContainerDied","Data":"b002fac7a0b8257754af315e1c00a446e1bedd6f8fa3355e65d0a0990baea7a4"}
Sep 16 19:56:52.632784 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.632529    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t" event={"ID":"d37e5925-ecd0-48a8-8beb-db0d2f5ab62a","Type":"ContainerDied","Data":"6b6184dcf0718ae1699031d210cba54290821a9c26427117cdf326a41a050884"}
Sep 16 19:56:52.632784 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.632553    2570 scope.go:117] "RemoveContainer" containerID="b002fac7a0b8257754af315e1c00a446e1bedd6f8fa3355e65d0a0990baea7a4"
Sep 16 19:56:52.643821 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.643767    2570 scope.go:117] "RemoveContainer" containerID="8b162c68a007b018b95afe620d1a7fda95e8eeeb34cb55307fdbfbbee7f145a2"
Sep 16 19:56:52.651658 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.651640    2570 scope.go:117] "RemoveContainer" containerID="b002fac7a0b8257754af315e1c00a446e1bedd6f8fa3355e65d0a0990baea7a4"
Sep 16 19:56:52.651947 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:56:52.651927    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b002fac7a0b8257754af315e1c00a446e1bedd6f8fa3355e65d0a0990baea7a4\": container with ID starting with b002fac7a0b8257754af315e1c00a446e1bedd6f8fa3355e65d0a0990baea7a4 not found: ID does not exist" containerID="b002fac7a0b8257754af315e1c00a446e1bedd6f8fa3355e65d0a0990baea7a4"
Sep 16 19:56:52.652006 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.651954    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b002fac7a0b8257754af315e1c00a446e1bedd6f8fa3355e65d0a0990baea7a4"} err="failed to get container status \"b002fac7a0b8257754af315e1c00a446e1bedd6f8fa3355e65d0a0990baea7a4\": rpc error: code = NotFound desc = could not find container \"b002fac7a0b8257754af315e1c00a446e1bedd6f8fa3355e65d0a0990baea7a4\": container with ID starting with b002fac7a0b8257754af315e1c00a446e1bedd6f8fa3355e65d0a0990baea7a4 not found: ID does not exist"
Sep 16 19:56:52.652006 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.651970    2570 scope.go:117] "RemoveContainer" containerID="8b162c68a007b018b95afe620d1a7fda95e8eeeb34cb55307fdbfbbee7f145a2"
Sep 16 19:56:52.652234 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:56:52.652216    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b162c68a007b018b95afe620d1a7fda95e8eeeb34cb55307fdbfbbee7f145a2\": container with ID starting with 8b162c68a007b018b95afe620d1a7fda95e8eeeb34cb55307fdbfbbee7f145a2 not found: ID does not exist" containerID="8b162c68a007b018b95afe620d1a7fda95e8eeeb34cb55307fdbfbbee7f145a2"
Sep 16 19:56:52.652284 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.652238    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b162c68a007b018b95afe620d1a7fda95e8eeeb34cb55307fdbfbbee7f145a2"} err="failed to get container status \"8b162c68a007b018b95afe620d1a7fda95e8eeeb34cb55307fdbfbbee7f145a2\": rpc error: code = NotFound desc = could not find container \"8b162c68a007b018b95afe620d1a7fda95e8eeeb34cb55307fdbfbbee7f145a2\": container with ID starting with 8b162c68a007b018b95afe620d1a7fda95e8eeeb34cb55307fdbfbbee7f145a2 not found: ID does not exist"
Sep 16 19:56:52.660503 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.660481    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"]
Sep 16 19:56:52.664742 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:52.664720    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7bc9c5d55-8xw6t"]
Sep 16 19:56:54.035979 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:54.035941    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f" podUID="7af11391-1a6f-44c5-afaf-f3270f96f34e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:54.160699 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:56:54.160647    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" path="/var/lib/kubelet/pods/d37e5925-ecd0-48a8-8beb-db0d2f5ab62a/volumes"
Sep 16 19:57:02.265176 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.265149    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"
Sep 16 19:57:02.357891 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.357816    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af11391-1a6f-44c5-afaf-f3270f96f34e-tls-certs\") pod \"7af11391-1a6f-44c5-afaf-f3270f96f34e\" (UID: \"7af11391-1a6f-44c5-afaf-f3270f96f34e\") "
Sep 16 19:57:02.357891 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.357878    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbjw6\" (UniqueName: \"kubernetes.io/projected/7af11391-1a6f-44c5-afaf-f3270f96f34e-kube-api-access-kbjw6\") pod \"7af11391-1a6f-44c5-afaf-f3270f96f34e\" (UID: \"7af11391-1a6f-44c5-afaf-f3270f96f34e\") "
Sep 16 19:57:02.359975 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.359942    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af11391-1a6f-44c5-afaf-f3270f96f34e-kube-api-access-kbjw6" (OuterVolumeSpecName: "kube-api-access-kbjw6") pod "7af11391-1a6f-44c5-afaf-f3270f96f34e" (UID: "7af11391-1a6f-44c5-afaf-f3270f96f34e"). InnerVolumeSpecName "kube-api-access-kbjw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:02.360079 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.359965    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af11391-1a6f-44c5-afaf-f3270f96f34e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7af11391-1a6f-44c5-afaf-f3270f96f34e" (UID: "7af11391-1a6f-44c5-afaf-f3270f96f34e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:57:02.459298 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.459269    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbjw6\" (UniqueName: \"kubernetes.io/projected/7af11391-1a6f-44c5-afaf-f3270f96f34e-kube-api-access-kbjw6\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:57:02.459298 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.459292    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af11391-1a6f-44c5-afaf-f3270f96f34e-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 19:57:02.668796 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.668725    2570 generic.go:356] "Generic (PLEG): container finished" podID="7af11391-1a6f-44c5-afaf-f3270f96f34e" containerID="42dbafab60a086e445791e593a3718e862bd08c064ad0440a8442b639f54ba05" exitCode=0
Sep 16 19:57:02.668796 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.668767    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"
Sep 16 19:57:02.668940 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.668768    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f" event={"ID":"7af11391-1a6f-44c5-afaf-f3270f96f34e","Type":"ContainerDied","Data":"42dbafab60a086e445791e593a3718e862bd08c064ad0440a8442b639f54ba05"}
Sep 16 19:57:02.668940 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.668908    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f" event={"ID":"7af11391-1a6f-44c5-afaf-f3270f96f34e","Type":"ContainerDied","Data":"b6da369994493046b7c5d4c9dcf61fed96665aed88224ba0db5e4dac87724434"}
Sep 16 19:57:02.669020 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.668940    2570 scope.go:117] "RemoveContainer" containerID="42dbafab60a086e445791e593a3718e862bd08c064ad0440a8442b639f54ba05"
Sep 16 19:57:02.677706 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.677674    2570 scope.go:117] "RemoveContainer" containerID="42dbafab60a086e445791e593a3718e862bd08c064ad0440a8442b639f54ba05"
Sep 16 19:57:02.677974 ip-10-0-135-244 kubenswrapper[2570]: E0916 19:57:02.677954    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42dbafab60a086e445791e593a3718e862bd08c064ad0440a8442b639f54ba05\": container with ID starting with 42dbafab60a086e445791e593a3718e862bd08c064ad0440a8442b639f54ba05 not found: ID does not exist" containerID="42dbafab60a086e445791e593a3718e862bd08c064ad0440a8442b639f54ba05"
Sep 16 19:57:02.678032 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.677981    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dbafab60a086e445791e593a3718e862bd08c064ad0440a8442b639f54ba05"} err="failed to get container status \"42dbafab60a086e445791e593a3718e862bd08c064ad0440a8442b639f54ba05\": rpc error: code = NotFound desc = could not find container \"42dbafab60a086e445791e593a3718e862bd08c064ad0440a8442b639f54ba05\": container with ID starting with 42dbafab60a086e445791e593a3718e862bd08c064ad0440a8442b639f54ba05 not found: ID does not exist"
Sep 16 19:57:02.691166 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.691144    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"]
Sep 16 19:57:02.697273 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:02.697252    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-v579f"]
Sep 16 19:57:04.160475 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:04.160442    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7af11391-1a6f-44c5-afaf-f3270f96f34e" path="/var/lib/kubelet/pods/7af11391-1a6f-44c5-afaf-f3270f96f34e/volumes"
Sep 16 19:57:25.895170 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.895134    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"]
Sep 16 19:57:25.895770 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.895366    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="storage-initializer"
Sep 16 19:57:25.895770 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.895377    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="storage-initializer"
Sep 16 19:57:25.895770 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.895389    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af11391-1a6f-44c5-afaf-f3270f96f34e" containerName="main"
Sep 16 19:57:25.895770 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.895397    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af11391-1a6f-44c5-afaf-f3270f96f34e" containerName="main"
Sep 16 19:57:25.895770 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.895410    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="main"
Sep 16 19:57:25.895770 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.895416    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="main"
Sep 16 19:57:25.895770 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.895459    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d37e5925-ecd0-48a8-8beb-db0d2f5ab62a" containerName="main"
Sep 16 19:57:25.895770 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.895466    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af11391-1a6f-44c5-afaf-f3270f96f34e" containerName="main"
Sep 16 19:57:25.898234 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.898215    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"
Sep 16 19:57:25.901775 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.901748    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 19:57:25.901775 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.901760    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-6t2dr\""
Sep 16 19:57:25.901935 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.901793    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-mbzfw\""
Sep 16 19:57:25.901935 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.901834    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 19:57:25.902093 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.902079    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 19:57:25.910679 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:25.910658    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"]
Sep 16 19:57:26.025348 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:26.025325    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2plw4\" (UniqueName: \"kubernetes.io/projected/e410a870-2ad2-499c-9fbb-29c71f406d8c-kube-api-access-2plw4\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67\" (UID: \"e410a870-2ad2-499c-9fbb-29c71f406d8c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"
Sep 16 19:57:26.025474 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:26.025357    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e410a870-2ad2-499c-9fbb-29c71f406d8c-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67\" (UID: \"e410a870-2ad2-499c-9fbb-29c71f406d8c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"
Sep 16 19:57:26.126234 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:26.126204    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2plw4\" (UniqueName: \"kubernetes.io/projected/e410a870-2ad2-499c-9fbb-29c71f406d8c-kube-api-access-2plw4\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67\" (UID: \"e410a870-2ad2-499c-9fbb-29c71f406d8c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"
Sep 16 19:57:26.126362 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:26.126241    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e410a870-2ad2-499c-9fbb-29c71f406d8c-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67\" (UID: \"e410a870-2ad2-499c-9fbb-29c71f406d8c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"
Sep 16 19:57:26.128636 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:26.128612    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e410a870-2ad2-499c-9fbb-29c71f406d8c-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67\" (UID: \"e410a870-2ad2-499c-9fbb-29c71f406d8c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"
Sep 16 19:57:26.135722 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:26.135701    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2plw4\" (UniqueName: \"kubernetes.io/projected/e410a870-2ad2-499c-9fbb-29c71f406d8c-kube-api-access-2plw4\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67\" (UID: \"e410a870-2ad2-499c-9fbb-29c71f406d8c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"
Sep 16 19:57:26.207768 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:26.207676    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"
Sep 16 19:57:26.505394 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:26.505305    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"]
Sep 16 19:57:26.508219 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:57:26.508192    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode410a870_2ad2_499c_9fbb_29c71f406d8c.slice/crio-f88ad3e8582b62e4407a6899ac88b90272c37f58680e5a6d69634fbeac993d34 WatchSource:0}: Error finding container f88ad3e8582b62e4407a6899ac88b90272c37f58680e5a6d69634fbeac993d34: Status 404 returned error can't find the container with id f88ad3e8582b62e4407a6899ac88b90272c37f58680e5a6d69634fbeac993d34
Sep 16 19:57:26.760461 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:26.760350    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67" event={"ID":"e410a870-2ad2-499c-9fbb-29c71f406d8c","Type":"ContainerStarted","Data":"e5e983626fe3b90e93756a560dc59db83fd92f4ccb306cb8771a3ba8dcbcd43e"}
Sep 16 19:57:26.760461 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:26.760425    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67" event={"ID":"e410a870-2ad2-499c-9fbb-29c71f406d8c","Type":"ContainerStarted","Data":"f88ad3e8582b62e4407a6899ac88b90272c37f58680e5a6d69634fbeac993d34"}
Sep 16 19:57:26.760935 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:26.760908    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"
Sep 16 19:57:57.767736 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:57.767639    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"
Sep 16 19:57:57.784736 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:57:57.784665    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67" podStartSLOduration=32.784650526 podStartE2EDuration="32.784650526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:26.800431531 +0000 UTC m=+4373.184990753" watchObservedRunningTime="2026-09-16 19:57:57.784650526 +0000 UTC m=+4404.169209750"
Sep 16 19:58:12.983776 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:12.983728    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"]
Sep 16 19:58:12.987020 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:12.987000    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:12.990530 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:12.990506    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-ttkd6\""
Sep 16 19:58:12.990679 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:12.990537    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:58:12.990810 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:12.990784    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 19:58:12.990810 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:12.990801    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 19:58:12.990810 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:12.990808    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 19:58:12.991038 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:12.990785    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 19:58:13.003459 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.003438    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"]
Sep 16 19:58:13.048799 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.048776    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.048892 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.048804    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f16bb75-d1e4-490e-8a04-8823686a6ac9-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.048892 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.048843    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f16bb75-d1e4-490e-8a04-8823686a6ac9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.048892 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.048879    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccw7r\" (UniqueName: \"kubernetes.io/projected/7f16bb75-d1e4-490e-8a04-8823686a6ac9-kube-api-access-ccw7r\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.048999 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.048927    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.048999 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.048948    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-home\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.048999 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.048965    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.048999 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.048983    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.149500 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.149480    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccw7r\" (UniqueName: \"kubernetes.io/projected/7f16bb75-d1e4-490e-8a04-8823686a6ac9-kube-api-access-ccw7r\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.149621 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.149515    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.149621 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.149537    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-home\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.149621 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.149554    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.149621 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.149573    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.149621 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.149591    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.150088 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.149648    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f16bb75-d1e4-490e-8a04-8823686a6ac9-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.150088 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.149734    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f16bb75-d1e4-490e-8a04-8823686a6ac9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.150088 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.149987    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-home\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.150088 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.150055    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.150229 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.150124    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.150229 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.150178    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.150542 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.150525    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f16bb75-d1e4-490e-8a04-8823686a6ac9-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.152040 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.152010    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.152243 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.152225    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f16bb75-d1e4-490e-8a04-8823686a6ac9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.158067 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.158046    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccw7r\" (UniqueName: \"kubernetes.io/projected/7f16bb75-d1e4-490e-8a04-8823686a6ac9-kube-api-access-ccw7r\") pod \"router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.297465 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.297443    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:13.586040 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.585864    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"]
Sep 16 19:58:13.588161 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:58:13.588133    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f16bb75_d1e4_490e_8a04_8823686a6ac9.slice/crio-12508adbaa6a335dff30a24dfb2dea216717f9ce7c02f1829a84405ba8a3b905 WatchSource:0}: Error finding container 12508adbaa6a335dff30a24dfb2dea216717f9ce7c02f1829a84405ba8a3b905: Status 404 returned error can't find the container with id 12508adbaa6a335dff30a24dfb2dea216717f9ce7c02f1829a84405ba8a3b905
Sep 16 19:58:13.929233 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.929147    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" event={"ID":"7f16bb75-d1e4-490e-8a04-8823686a6ac9","Type":"ContainerStarted","Data":"e422f771b8339fffa90e613be549fd95c51410f37acb63baf9768aefc1a3a9b5"}
Sep 16 19:58:13.929233 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:13.929205    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" event={"ID":"7f16bb75-d1e4-490e-8a04-8823686a6ac9","Type":"ContainerStarted","Data":"12508adbaa6a335dff30a24dfb2dea216717f9ce7c02f1829a84405ba8a3b905"}
Sep 16 19:58:15.937055 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:15.937021    2570 generic.go:356] "Generic (PLEG): container finished" podID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerID="e422f771b8339fffa90e613be549fd95c51410f37acb63baf9768aefc1a3a9b5" exitCode=0
Sep 16 19:58:15.937397 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:15.937094    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" event={"ID":"7f16bb75-d1e4-490e-8a04-8823686a6ac9","Type":"ContainerDied","Data":"e422f771b8339fffa90e613be549fd95c51410f37acb63baf9768aefc1a3a9b5"}
Sep 16 19:58:16.943189 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:16.943155    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" event={"ID":"7f16bb75-d1e4-490e-8a04-8823686a6ac9","Type":"ContainerStarted","Data":"2d6ca6e91b82f4e13c8786bc407e868730c553d806320d125833cc40ed4fa27f"}
Sep 16 19:58:16.967909 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:16.967859    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" podStartSLOduration=2.839165372 podStartE2EDuration="4.967843001s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.128677629s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:58:16.966199114 +0000 UTC m=+4423.350758337" watchObservedRunningTime="2026-09-16 19:58:16.967843001 +0000 UTC m=+4423.352402224"
Sep 16 19:58:23.297892 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:23.297851    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:23.298435 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:23.297926    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 19:58:23.299351 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:23.299320    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:58:33.298826 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:33.298785    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:58:43.298833 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:43.298788    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:58:53.298353 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:58:53.298306    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:59:03.298405 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:03.298356    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:59:13.298226 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:13.298184    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:59:23.298518 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:23.298470    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:59:28.112015 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.111979    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"]
Sep 16 19:59:28.114140 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.114111    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.116758 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.116727    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 19:59:28.116903 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.116763    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:59:28.129280 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.129253    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"]
Sep 16 19:59:28.232452 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.232422    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-tls-certs\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.232452 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.232455    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-dshm\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.232670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.232472    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b497v\" (UniqueName: \"kubernetes.io/projected/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-kube-api-access-b497v\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.232670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.232513    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-home\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.232670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.232550    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-cabundle-cert\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.232670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.232584    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-tmp-dir\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.232670 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.232644    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-model-cache\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.232955 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.232736    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-kserve-provision-location\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.333384 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.333350    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-model-cache\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.333566 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.333398    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-kserve-provision-location\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.333566 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.333507    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-tls-certs\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.333566 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.333527    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-dshm\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.333566 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.333542    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b497v\" (UniqueName: \"kubernetes.io/projected/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-kube-api-access-b497v\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.333566 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.333557    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-home\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.333809 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.333608    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-cabundle-cert\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.333809 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.333652    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-tmp-dir\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.333928 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.333904    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-model-cache\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.333985 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.333937    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-kserve-provision-location\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.334052 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.333980    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-tmp-dir\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.334756 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.334731    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-home\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.335285 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.335257    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-cabundle-cert\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.336218 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.336194    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-dshm\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.343819 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.343789    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-tls-certs\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.344405 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.344383    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b497v\" (UniqueName: \"kubernetes.io/projected/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-kube-api-access-b497v\") pod \"prestop-hook-test-kserve-685bf59966-fhknb\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.424152 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.424053    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:28.710814 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:28.710750    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"]
Sep 16 19:59:28.713266 ip-10-0-135-244 kubenswrapper[2570]: W0916 19:59:28.713237    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e81d1f4_1c93_4b8d_a7fb_eec0b3faf8fd.slice/crio-b829dab92fc1d0c959a304c6bded1bc9145eb18445b6fdd11e97681bb445ca71 WatchSource:0}: Error finding container b829dab92fc1d0c959a304c6bded1bc9145eb18445b6fdd11e97681bb445ca71: Status 404 returned error can't find the container with id b829dab92fc1d0c959a304c6bded1bc9145eb18445b6fdd11e97681bb445ca71
Sep 16 19:59:29.226281 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:29.226233    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb" event={"ID":"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd","Type":"ContainerStarted","Data":"db5cf79effd60e3c9b7cc65f95aa3d78733c03b8b3f5e26b2bee4f0c88f0a0e5"}
Sep 16 19:59:29.226788 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:29.226301    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb" event={"ID":"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd","Type":"ContainerStarted","Data":"b829dab92fc1d0c959a304c6bded1bc9145eb18445b6fdd11e97681bb445ca71"}
Sep 16 19:59:31.235371 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:31.235281    2570 generic.go:356] "Generic (PLEG): container finished" podID="3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" containerID="db5cf79effd60e3c9b7cc65f95aa3d78733c03b8b3f5e26b2bee4f0c88f0a0e5" exitCode=0
Sep 16 19:59:31.235371 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:31.235342    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb" event={"ID":"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd","Type":"ContainerDied","Data":"db5cf79effd60e3c9b7cc65f95aa3d78733c03b8b3f5e26b2bee4f0c88f0a0e5"}
Sep 16 19:59:32.241983 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:32.241944    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb" event={"ID":"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd","Type":"ContainerStarted","Data":"92272286b8baae7f492349ce787a873d1f194308c5498abfbee9813cfe1036f6"}
Sep 16 19:59:32.269058 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:32.268997    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb" podStartSLOduration=2.072210943 podStartE2EDuration="4.268981295s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.196770352s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:59:32.26670661 +0000 UTC m=+4498.651265825" watchObservedRunningTime="2026-09-16 19:59:32.268981295 +0000 UTC m=+4498.653540517"
Sep 16 19:59:32.425082 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:32.425040    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:33.298457 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:33.298410    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:59:38.424428 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:38.424392    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 19:59:38.426122 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:38.426093    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb" podUID="3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:59:43.298992 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:43.298943    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:59:48.424905 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:48.424860    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb" podUID="3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 16 19:59:50.309325 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:50.309291    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"]
Sep 16 19:59:50.318001 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:50.317973    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"]
Sep 16 19:59:53.298787 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:53.298741    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 19:59:57.767783 ip-10-0-135-244 kubenswrapper[2570]: I0916 19:59:57.767735    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67" podUID="e410a870-2ad2-499c-9fbb-29c71f406d8c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:03.298900 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:03.298854    2570 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Sep 16 20:00:05.335787 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:05.335745    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb" podUID="3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" containerName="main" containerID="cri-o://92272286b8baae7f492349ce787a873d1f194308c5498abfbee9813cfe1036f6" gracePeriod=45
Sep 16 20:00:05.341825 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:05.341789    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67" podUID="e410a870-2ad2-499c-9fbb-29c71f406d8c" containerName="main" containerID="cri-o://e5e983626fe3b90e93756a560dc59db83fd92f4ccb306cb8771a3ba8dcbcd43e" gracePeriod=45
Sep 16 20:00:07.766807 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:07.766763    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67" podUID="e410a870-2ad2-499c-9fbb-29c71f406d8c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:13.308570 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:13.308535    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 20:00:13.316137 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:13.316112    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 20:00:17.767743 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:17.767681    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67" podUID="e410a870-2ad2-499c-9fbb-29c71f406d8c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:17.768228 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:17.767832    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"
Sep 16 20:00:22.880746 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:22.880706    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"]
Sep 16 20:00:27.766972 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:27.766931    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67" podUID="e410a870-2ad2-499c-9fbb-29c71f406d8c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:35.512935 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:35.512905    2570 generic.go:356] "Generic (PLEG): container finished" podID="e410a870-2ad2-499c-9fbb-29c71f406d8c" containerID="e5e983626fe3b90e93756a560dc59db83fd92f4ccb306cb8771a3ba8dcbcd43e" exitCode=0
Sep 16 20:00:35.513259 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:35.512954    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67" event={"ID":"e410a870-2ad2-499c-9fbb-29c71f406d8c","Type":"ContainerDied","Data":"e5e983626fe3b90e93756a560dc59db83fd92f4ccb306cb8771a3ba8dcbcd43e"}
Sep 16 20:00:35.592664 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:35.592632    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"
Sep 16 20:00:35.703359 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:35.703272    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e410a870-2ad2-499c-9fbb-29c71f406d8c-tls-certs\") pod \"e410a870-2ad2-499c-9fbb-29c71f406d8c\" (UID: \"e410a870-2ad2-499c-9fbb-29c71f406d8c\") "
Sep 16 20:00:35.703517 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:35.703379    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2plw4\" (UniqueName: \"kubernetes.io/projected/e410a870-2ad2-499c-9fbb-29c71f406d8c-kube-api-access-2plw4\") pod \"e410a870-2ad2-499c-9fbb-29c71f406d8c\" (UID: \"e410a870-2ad2-499c-9fbb-29c71f406d8c\") "
Sep 16 20:00:35.705479 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:35.705446    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e410a870-2ad2-499c-9fbb-29c71f406d8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e410a870-2ad2-499c-9fbb-29c71f406d8c" (UID: "e410a870-2ad2-499c-9fbb-29c71f406d8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:00:35.705583 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:35.705538    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e410a870-2ad2-499c-9fbb-29c71f406d8c-kube-api-access-2plw4" (OuterVolumeSpecName: "kube-api-access-2plw4") pod "e410a870-2ad2-499c-9fbb-29c71f406d8c" (UID: "e410a870-2ad2-499c-9fbb-29c71f406d8c"). InnerVolumeSpecName "kube-api-access-2plw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:00:35.804353 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:35.804319    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2plw4\" (UniqueName: \"kubernetes.io/projected/e410a870-2ad2-499c-9fbb-29c71f406d8c-kube-api-access-2plw4\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.804353 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:35.804352    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e410a870-2ad2-499c-9fbb-29c71f406d8c-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:36.518601 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:36.518573    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"
Sep 16 20:00:36.519077 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:36.518598    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67" event={"ID":"e410a870-2ad2-499c-9fbb-29c71f406d8c","Type":"ContainerDied","Data":"f88ad3e8582b62e4407a6899ac88b90272c37f58680e5a6d69634fbeac993d34"}
Sep 16 20:00:36.519077 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:36.518653    2570 scope.go:117] "RemoveContainer" containerID="e5e983626fe3b90e93756a560dc59db83fd92f4ccb306cb8771a3ba8dcbcd43e"
Sep 16 20:00:36.534634 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:36.534610    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"]
Sep 16 20:00:36.543139 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:36.543113    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-l4z67"]
Sep 16 20:00:37.909410 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:37.909366    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main" containerID="cri-o://2d6ca6e91b82f4e13c8786bc407e868730c553d806320d125833cc40ed4fa27f" gracePeriod=45
Sep 16 20:00:38.161721 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.161621    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e410a870-2ad2-499c-9fbb-29c71f406d8c" path="/var/lib/kubelet/pods/e410a870-2ad2-499c-9fbb-29c71f406d8c/volumes"
Sep 16 20:00:38.530409 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.530378    2570 generic.go:356] "Generic (PLEG): container finished" podID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerID="2d6ca6e91b82f4e13c8786bc407e868730c553d806320d125833cc40ed4fa27f" exitCode=0
Sep 16 20:00:38.530522 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.530454    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" event={"ID":"7f16bb75-d1e4-490e-8a04-8823686a6ac9","Type":"ContainerDied","Data":"2d6ca6e91b82f4e13c8786bc407e868730c553d806320d125833cc40ed4fa27f"}
Sep 16 20:00:38.651435 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.651404    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 20:00:38.731837 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.731765    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f16bb75-d1e4-490e-8a04-8823686a6ac9-cabundle-cert\") pod \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") "
Sep 16 20:00:38.731837 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.731798    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-model-cache\") pod \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") "
Sep 16 20:00:38.731837 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.731819    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-kserve-provision-location\") pod \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") "
Sep 16 20:00:38.732095 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.731852    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-home\") pod \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") "
Sep 16 20:00:38.732095 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.731879    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccw7r\" (UniqueName: \"kubernetes.io/projected/7f16bb75-d1e4-490e-8a04-8823686a6ac9-kube-api-access-ccw7r\") pod \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") "
Sep 16 20:00:38.732095 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.731924    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f16bb75-d1e4-490e-8a04-8823686a6ac9-tls-certs\") pod \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") "
Sep 16 20:00:38.732095 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.731966    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-tmp-dir\") pod \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") "
Sep 16 20:00:38.732095 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.732009    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-dshm\") pod \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\" (UID: \"7f16bb75-d1e4-490e-8a04-8823686a6ac9\") "
Sep 16 20:00:38.732361 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.732133    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f16bb75-d1e4-490e-8a04-8823686a6ac9" (UID: "7f16bb75-d1e4-490e-8a04-8823686a6ac9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:38.732361 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.732198    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f16bb75-d1e4-490e-8a04-8823686a6ac9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f16bb75-d1e4-490e-8a04-8823686a6ac9" (UID: "7f16bb75-d1e4-490e-8a04-8823686a6ac9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:00:38.732361 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.732241    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:38.732513 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.732454    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-home" (OuterVolumeSpecName: "home") pod "7f16bb75-d1e4-490e-8a04-8823686a6ac9" (UID: "7f16bb75-d1e4-490e-8a04-8823686a6ac9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:38.734418 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.734385    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-dshm" (OuterVolumeSpecName: "dshm") pod "7f16bb75-d1e4-490e-8a04-8823686a6ac9" (UID: "7f16bb75-d1e4-490e-8a04-8823686a6ac9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:38.734545 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.734457    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f16bb75-d1e4-490e-8a04-8823686a6ac9-kube-api-access-ccw7r" (OuterVolumeSpecName: "kube-api-access-ccw7r") pod "7f16bb75-d1e4-490e-8a04-8823686a6ac9" (UID: "7f16bb75-d1e4-490e-8a04-8823686a6ac9"). InnerVolumeSpecName "kube-api-access-ccw7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:00:38.734545 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.734531    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f16bb75-d1e4-490e-8a04-8823686a6ac9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f16bb75-d1e4-490e-8a04-8823686a6ac9" (UID: "7f16bb75-d1e4-490e-8a04-8823686a6ac9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:00:38.752393 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.752362    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f16bb75-d1e4-490e-8a04-8823686a6ac9" (UID: "7f16bb75-d1e4-490e-8a04-8823686a6ac9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:38.754504 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.754482    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f16bb75-d1e4-490e-8a04-8823686a6ac9" (UID: "7f16bb75-d1e4-490e-8a04-8823686a6ac9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:38.832737 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.832709    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f16bb75-d1e4-490e-8a04-8823686a6ac9-cabundle-cert\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:38.832737 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.832736    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-kserve-provision-location\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:38.832876 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.832746    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:38.832876 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.832755    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccw7r\" (UniqueName: \"kubernetes.io/projected/7f16bb75-d1e4-490e-8a04-8823686a6ac9-kube-api-access-ccw7r\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:38.832876 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.832764    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f16bb75-d1e4-490e-8a04-8823686a6ac9-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:38.832876 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.832772    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:38.832876 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:38.832781    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f16bb75-d1e4-490e-8a04-8823686a6ac9-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:39.535479 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:39.535444    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4" event={"ID":"7f16bb75-d1e4-490e-8a04-8823686a6ac9","Type":"ContainerDied","Data":"12508adbaa6a335dff30a24dfb2dea216717f9ce7c02f1829a84405ba8a3b905"}
Sep 16 20:00:39.535880 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:39.535501    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"
Sep 16 20:00:39.535880 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:39.535508    2570 scope.go:117] "RemoveContainer" containerID="2d6ca6e91b82f4e13c8786bc407e868730c553d806320d125833cc40ed4fa27f"
Sep 16 20:00:39.549271 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:39.549014    2570 scope.go:117] "RemoveContainer" containerID="e422f771b8339fffa90e613be549fd95c51410f37acb63baf9768aefc1a3a9b5"
Sep 16 20:00:39.572428 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:39.572401    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"]
Sep 16 20:00:39.592065 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:39.592040    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-6c66874d65-t96l4"]
Sep 16 20:00:40.161883 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:40.161847    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" path="/var/lib/kubelet/pods/7f16bb75-d1e4-490e-8a04-8823686a6ac9/volumes"
Sep 16 20:00:50.581576 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:50.581548    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-685bf59966-fhknb_3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd/main/0.log"
Sep 16 20:00:50.582002 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:50.581980    2570 generic.go:356] "Generic (PLEG): container finished" podID="3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" containerID="92272286b8baae7f492349ce787a873d1f194308c5498abfbee9813cfe1036f6" exitCode=137
Sep 16 20:00:50.582060 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:50.582047    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb" event={"ID":"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd","Type":"ContainerDied","Data":"92272286b8baae7f492349ce787a873d1f194308c5498abfbee9813cfe1036f6"}
Sep 16 20:00:50.966816 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:50.966793    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-685bf59966-fhknb_3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd/main/0.log"
Sep 16 20:00:50.969432 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:50.969415    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 20:00:51.126913 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.126838    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-home\") pod \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") "
Sep 16 20:00:51.126913 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.126883    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b497v\" (UniqueName: \"kubernetes.io/projected/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-kube-api-access-b497v\") pod \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") "
Sep 16 20:00:51.126913 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.126913    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-tls-certs\") pod \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") "
Sep 16 20:00:51.127185 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.126928    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-tmp-dir\") pod \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") "
Sep 16 20:00:51.127185 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.126952    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-model-cache\") pod \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") "
Sep 16 20:00:51.127185 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.126968    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-cabundle-cert\") pod \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") "
Sep 16 20:00:51.127185 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.126995    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-kserve-provision-location\") pod \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") "
Sep 16 20:00:51.127185 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.127033    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-dshm\") pod \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\" (UID: \"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd\") "
Sep 16 20:00:51.127432 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.127300    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-home" (OuterVolumeSpecName: "home") pod "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" (UID: "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:51.127525 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.127499    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" (UID: "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:51.127585 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.127543    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" (UID: "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:00:51.127683 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.127654    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" (UID: "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:51.129053 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.129029    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-kube-api-access-b497v" (OuterVolumeSpecName: "kube-api-access-b497v") pod "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" (UID: "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd"). InnerVolumeSpecName "kube-api-access-b497v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:00:51.129208 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.129191    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" (UID: "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:00:51.129461 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.129440    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-dshm" (OuterVolumeSpecName: "dshm") pod "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" (UID: "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:51.146539 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.146511    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" (UID: "3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:51.228422 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.228398    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:51.228422 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.228417    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:51.228545 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.228426    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b497v\" (UniqueName: \"kubernetes.io/projected/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-kube-api-access-b497v\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:51.228545 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.228435    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:51.228545 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.228445    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:51.228545 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.228453    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:51.228545 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.228461    2570 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-cabundle-cert\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:51.228545 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.228468    2570 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd-kserve-provision-location\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:00:51.587241 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.587217    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-685bf59966-fhknb_3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd/main/0.log"
Sep 16 20:00:51.587702 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.587666    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"
Sep 16 20:00:51.587783 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.587666    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb" event={"ID":"3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd","Type":"ContainerDied","Data":"b829dab92fc1d0c959a304c6bded1bc9145eb18445b6fdd11e97681bb445ca71"}
Sep 16 20:00:51.587838 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.587825    2570 scope.go:117] "RemoveContainer" containerID="92272286b8baae7f492349ce787a873d1f194308c5498abfbee9813cfe1036f6"
Sep 16 20:00:51.595958 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.595938    2570 scope.go:117] "RemoveContainer" containerID="db5cf79effd60e3c9b7cc65f95aa3d78733c03b8b3f5e26b2bee4f0c88f0a0e5"
Sep 16 20:00:51.614349 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.614326    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"]
Sep 16 20:00:51.621553 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:51.621533    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-685bf59966-fhknb"]
Sep 16 20:00:52.160803 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:00:52.160770    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" path="/var/lib/kubelet/pods/3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd/volumes"
Sep 16 20:01:05.358004 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.357924    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"]
Sep 16 20:01:05.358358 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.358292    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="storage-initializer"
Sep 16 20:01:05.358358 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.358309    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="storage-initializer"
Sep 16 20:01:05.358358 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.358324    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e410a870-2ad2-499c-9fbb-29c71f406d8c" containerName="main"
Sep 16 20:01:05.358358 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.358332    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e410a870-2ad2-499c-9fbb-29c71f406d8c" containerName="main"
Sep 16 20:01:05.358358 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.358347    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main"
Sep 16 20:01:05.358358 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.358356    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main"
Sep 16 20:01:05.358558 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.358365    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" containerName="main"
Sep 16 20:01:05.358558 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.358374    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" containerName="main"
Sep 16 20:01:05.358558 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.358387    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" containerName="storage-initializer"
Sep 16 20:01:05.358558 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.358395    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" containerName="storage-initializer"
Sep 16 20:01:05.358558 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.358456    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f16bb75-d1e4-490e-8a04-8823686a6ac9" containerName="main"
Sep 16 20:01:05.358558 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.358468    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e81d1f4-1c93-4b8d-a7fb-eec0b3faf8fd" containerName="main"
Sep 16 20:01:05.358558 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.358478    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e410a870-2ad2-499c-9fbb-29c71f406d8c" containerName="main"
Sep 16 20:01:05.361333 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.361313    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.369345 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.369323    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 20:01:05.369464 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.369323    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-pxvng\""
Sep 16 20:01:05.369464 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.369323    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 20:01:05.369596 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.369328    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 20:01:05.381671 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.381649    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"]
Sep 16 20:01:05.421993 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.421965    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj2px\" (UniqueName: \"kubernetes.io/projected/b1e57bb7-3360-4362-b53b-119f5dcdc56c-kube-api-access-rj2px\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.422104 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.421995    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.422104 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.422015    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.422104 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.422043    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e57bb7-3360-4362-b53b-119f5dcdc56c-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.422251 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.422112    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.422251 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.422164    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.522946 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.522912    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj2px\" (UniqueName: \"kubernetes.io/projected/b1e57bb7-3360-4362-b53b-119f5dcdc56c-kube-api-access-rj2px\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.522946 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.522952    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.523089 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.522971    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.523089 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.522995    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e57bb7-3360-4362-b53b-119f5dcdc56c-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.523089 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.523039    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.523089 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.523077    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.523359 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.523338    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.523440 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.523379    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.523440 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.523431    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.525167 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.525141    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.525421 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.525405    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e57bb7-3360-4362-b53b-119f5dcdc56c-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.531414 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.531393    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj2px\" (UniqueName: \"kubernetes.io/projected/b1e57bb7-3360-4362-b53b-119f5dcdc56c-kube-api-access-rj2px\") pod \"rolling-upgrade-test-kserve-74b79d7f77-snwjw\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.671756 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.671673    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:05.940159 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:05.940131    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"]
Sep 16 20:01:05.942622 ip-10-0-135-244 kubenswrapper[2570]: W0916 20:01:05.942593    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e57bb7_3360_4362_b53b_119f5dcdc56c.slice/crio-aa91abc31b65c5f75ddd9bb746a43e71fec89307e4474c0efc062d667c837aac WatchSource:0}: Error finding container aa91abc31b65c5f75ddd9bb746a43e71fec89307e4474c0efc062d667c837aac: Status 404 returned error can't find the container with id aa91abc31b65c5f75ddd9bb746a43e71fec89307e4474c0efc062d667c837aac
Sep 16 20:01:06.643932 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:06.643885    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw" event={"ID":"b1e57bb7-3360-4362-b53b-119f5dcdc56c","Type":"ContainerStarted","Data":"4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1"}
Sep 16 20:01:06.644531 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:06.643958    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw" event={"ID":"b1e57bb7-3360-4362-b53b-119f5dcdc56c","Type":"ContainerStarted","Data":"aa91abc31b65c5f75ddd9bb746a43e71fec89307e4474c0efc062d667c837aac"}
Sep 16 20:01:06.669578 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:06.669525    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw" podStartSLOduration=1.669509885 podStartE2EDuration="1.669509885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:01:06.66884369 +0000 UTC m=+4593.053402915" watchObservedRunningTime="2026-09-16 20:01:06.669509885 +0000 UTC m=+4593.054069107"
Sep 16 20:01:06.672051 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:06.672023    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:15.672049 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:15.672012    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:15.684809 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:15.684776    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:15.695906 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:15.695884    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:52.049728 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.049694    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"]
Sep 16 20:01:52.058085 ip-10-0-135-244 kubenswrapper[2570]: E0916 20:01:52.058043    2570 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:01:52.058085 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:01:52.058085 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 20:01:52.058085 ip-10-0-135-244 kubenswrapper[2570]:  > containerID="4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1" cmd=["/bin/sleep","15"]
Sep 16 20:01:52.058310 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.058102    2570 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw" message="" err=<
Sep 16 20:01:52.058310 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:01:52.058310 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 20:01:52.058310 ip-10-0-135-244 kubenswrapper[2570]:  >
Sep 16 20:01:52.058310 ip-10-0-135-244 kubenswrapper[2570]: E0916 20:01:52.058114    2570 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:01:52.058310 ip-10-0-135-244 kubenswrapper[2570]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:01:52.058310 ip-10-0-135-244 kubenswrapper[2570]:         , stdout: , stderr: , exit code -1
Sep 16 20:01:52.058310 ip-10-0-135-244 kubenswrapper[2570]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw" podUID="b1e57bb7-3360-4362-b53b-119f5dcdc56c" containerName="main" containerID="cri-o://4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1"
Sep 16 20:01:52.058310 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.058162    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw" podUID="b1e57bb7-3360-4362-b53b-119f5dcdc56c" containerName="main" containerID="cri-o://4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1" gracePeriod=60
Sep 16 20:01:52.388003 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.387980    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:52.449477 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.449445    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-tmp-dir\") pod \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") "
Sep 16 20:01:52.449636 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.449489    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e57bb7-3360-4362-b53b-119f5dcdc56c-tls-certs\") pod \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") "
Sep 16 20:01:52.449636 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.449513    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-home\") pod \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") "
Sep 16 20:01:52.449636 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.449528    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-dshm\") pod \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") "
Sep 16 20:01:52.449636 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.449556    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj2px\" (UniqueName: \"kubernetes.io/projected/b1e57bb7-3360-4362-b53b-119f5dcdc56c-kube-api-access-rj2px\") pod \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") "
Sep 16 20:01:52.449636 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.449575    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-model-cache\") pod \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\" (UID: \"b1e57bb7-3360-4362-b53b-119f5dcdc56c\") "
Sep 16 20:01:52.449914 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.449768    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1e57bb7-3360-4362-b53b-119f5dcdc56c" (UID: "b1e57bb7-3360-4362-b53b-119f5dcdc56c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:52.449914 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.449783    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-home" (OuterVolumeSpecName: "home") pod "b1e57bb7-3360-4362-b53b-119f5dcdc56c" (UID: "b1e57bb7-3360-4362-b53b-119f5dcdc56c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:52.450037 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.449929    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1e57bb7-3360-4362-b53b-119f5dcdc56c" (UID: "b1e57bb7-3360-4362-b53b-119f5dcdc56c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:52.451793 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.451771    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-dshm" (OuterVolumeSpecName: "dshm") pod "b1e57bb7-3360-4362-b53b-119f5dcdc56c" (UID: "b1e57bb7-3360-4362-b53b-119f5dcdc56c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:52.451884 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.451840    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e57bb7-3360-4362-b53b-119f5dcdc56c-kube-api-access-rj2px" (OuterVolumeSpecName: "kube-api-access-rj2px") pod "b1e57bb7-3360-4362-b53b-119f5dcdc56c" (UID: "b1e57bb7-3360-4362-b53b-119f5dcdc56c"). InnerVolumeSpecName "kube-api-access-rj2px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:52.451884 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.451860    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e57bb7-3360-4362-b53b-119f5dcdc56c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1e57bb7-3360-4362-b53b-119f5dcdc56c" (UID: "b1e57bb7-3360-4362-b53b-119f5dcdc56c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:01:52.550866 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.550822    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-tmp-dir\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:01:52.550866 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.550859    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1e57bb7-3360-4362-b53b-119f5dcdc56c-tls-certs\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:01:52.550866 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.550868    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-home\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:01:52.550866 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.550876    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-dshm\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:01:52.550866 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.550884    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj2px\" (UniqueName: \"kubernetes.io/projected/b1e57bb7-3360-4362-b53b-119f5dcdc56c-kube-api-access-rj2px\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:01:52.551165 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.550893    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1e57bb7-3360-4362-b53b-119f5dcdc56c-model-cache\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:01:52.812437 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.812398    2570 generic.go:356] "Generic (PLEG): container finished" podID="b1e57bb7-3360-4362-b53b-119f5dcdc56c" containerID="4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1" exitCode=0
Sep 16 20:01:52.812624 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.812447    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"
Sep 16 20:01:52.812624 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.812489    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw" event={"ID":"b1e57bb7-3360-4362-b53b-119f5dcdc56c","Type":"ContainerDied","Data":"4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1"}
Sep 16 20:01:52.812624 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.812535    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw" event={"ID":"b1e57bb7-3360-4362-b53b-119f5dcdc56c","Type":"ContainerDied","Data":"aa91abc31b65c5f75ddd9bb746a43e71fec89307e4474c0efc062d667c837aac"}
Sep 16 20:01:52.812624 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.812558    2570 scope.go:117] "RemoveContainer" containerID="4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1"
Sep 16 20:01:52.822554 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.822535    2570 scope.go:117] "RemoveContainer" containerID="4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1"
Sep 16 20:01:52.822950 ip-10-0-135-244 kubenswrapper[2570]: E0916 20:01:52.822880    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1\": container with ID starting with 4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1 not found: ID does not exist" containerID="4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1"
Sep 16 20:01:52.822950 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.822922    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1"} err="failed to get container status \"4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1\": rpc error: code = NotFound desc = could not find container \"4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1\": container with ID starting with 4b6e0be1ee2697dde155b3d1dac14e5cf389ed9b9535a3630316e7f146547fb1 not found: ID does not exist"
Sep 16 20:01:52.861737 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.861709    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"]
Sep 16 20:01:52.875701 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:52.875663    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-snwjw"]
Sep 16 20:01:54.160139 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:01:54.160103    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e57bb7-3360-4362-b53b-119f5dcdc56c" path="/var/lib/kubelet/pods/b1e57bb7-3360-4362-b53b-119f5dcdc56c/volumes"
Sep 16 20:04:03.071548 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.071510    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mht7h/must-gather-mc2nt"]
Sep 16 20:04:03.071957 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.071766    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e57bb7-3360-4362-b53b-119f5dcdc56c" containerName="main"
Sep 16 20:04:03.071957 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.071777    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e57bb7-3360-4362-b53b-119f5dcdc56c" containerName="main"
Sep 16 20:04:03.071957 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.071822    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e57bb7-3360-4362-b53b-119f5dcdc56c" containerName="main"
Sep 16 20:04:03.074521 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.074505    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mht7h/must-gather-mc2nt"
Sep 16 20:04:03.077613 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.077594    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mht7h\"/\"kube-root-ca.crt\""
Sep 16 20:04:03.078375 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.078356    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mht7h\"/\"default-dockercfg-2btdq\""
Sep 16 20:04:03.079213 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.079192    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mht7h\"/\"openshift-service-ca.crt\""
Sep 16 20:04:03.090897 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.090876    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mht7h/must-gather-mc2nt"]
Sep 16 20:04:03.138283 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.138257    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8c16109-e449-49aa-9a22-68b307fe6151-must-gather-output\") pod \"must-gather-mc2nt\" (UID: \"d8c16109-e449-49aa-9a22-68b307fe6151\") " pod="openshift-must-gather-mht7h/must-gather-mc2nt"
Sep 16 20:04:03.138380 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.138283    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvjq7\" (UniqueName: \"kubernetes.io/projected/d8c16109-e449-49aa-9a22-68b307fe6151-kube-api-access-mvjq7\") pod \"must-gather-mc2nt\" (UID: \"d8c16109-e449-49aa-9a22-68b307fe6151\") " pod="openshift-must-gather-mht7h/must-gather-mc2nt"
Sep 16 20:04:03.239268 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.239241    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8c16109-e449-49aa-9a22-68b307fe6151-must-gather-output\") pod \"must-gather-mc2nt\" (UID: \"d8c16109-e449-49aa-9a22-68b307fe6151\") " pod="openshift-must-gather-mht7h/must-gather-mc2nt"
Sep 16 20:04:03.239268 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.239268    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvjq7\" (UniqueName: \"kubernetes.io/projected/d8c16109-e449-49aa-9a22-68b307fe6151-kube-api-access-mvjq7\") pod \"must-gather-mc2nt\" (UID: \"d8c16109-e449-49aa-9a22-68b307fe6151\") " pod="openshift-must-gather-mht7h/must-gather-mc2nt"
Sep 16 20:04:03.239590 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.239570    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8c16109-e449-49aa-9a22-68b307fe6151-must-gather-output\") pod \"must-gather-mc2nt\" (UID: \"d8c16109-e449-49aa-9a22-68b307fe6151\") " pod="openshift-must-gather-mht7h/must-gather-mc2nt"
Sep 16 20:04:03.248748 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.248719    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvjq7\" (UniqueName: \"kubernetes.io/projected/d8c16109-e449-49aa-9a22-68b307fe6151-kube-api-access-mvjq7\") pod \"must-gather-mc2nt\" (UID: \"d8c16109-e449-49aa-9a22-68b307fe6151\") " pod="openshift-must-gather-mht7h/must-gather-mc2nt"
Sep 16 20:04:03.383213 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.383160    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mht7h/must-gather-mc2nt"
Sep 16 20:04:03.651104 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.651078    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mht7h/must-gather-mc2nt"]
Sep 16 20:04:03.653537 ip-10-0-135-244 kubenswrapper[2570]: W0916 20:04:03.653509    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8c16109_e449_49aa_9a22_68b307fe6151.slice/crio-7793635310e35d987e0c44c802e71353497fcb2bdbf4f34f401ed5b04832b0f8 WatchSource:0}: Error finding container 7793635310e35d987e0c44c802e71353497fcb2bdbf4f34f401ed5b04832b0f8: Status 404 returned error can't find the container with id 7793635310e35d987e0c44c802e71353497fcb2bdbf4f34f401ed5b04832b0f8
Sep 16 20:04:03.655433 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:03.655415    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:04:04.263137 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:04.263100    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mht7h/must-gather-mc2nt" event={"ID":"d8c16109-e449-49aa-9a22-68b307fe6151","Type":"ContainerStarted","Data":"7793635310e35d987e0c44c802e71353497fcb2bdbf4f34f401ed5b04832b0f8"}
Sep 16 20:04:10.291558 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:10.291507    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mht7h/must-gather-mc2nt" event={"ID":"d8c16109-e449-49aa-9a22-68b307fe6151","Type":"ContainerStarted","Data":"fa5148b2b0dff3c54f548132f3290027eba368526189c4a33cb966530ea4e20b"}
Sep 16 20:04:10.291558 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:10.291563    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mht7h/must-gather-mc2nt" event={"ID":"d8c16109-e449-49aa-9a22-68b307fe6151","Type":"ContainerStarted","Data":"d437b341c24eb80b43bbe184ce2782b3d878515a88db5f9db0d90a9ce3a1062e"}
Sep 16 20:04:10.308398 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:10.308348    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mht7h/must-gather-mc2nt" podStartSLOduration=1.275703723 podStartE2EDuration="7.308330824s" totalImagesPullingTime="6.032627101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:04:10.306637088 +0000 UTC m=+4776.691196313" watchObservedRunningTime="2026-09-16 20:04:10.308330824 +0000 UTC m=+4776.692890047"
Sep 16 20:04:26.111661 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:26.111628    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j_d7508170-184a-47cc-9319-c53b95dccd82/istio-proxy/0.log"
Sep 16 20:04:26.128476 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:26.128446    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j_d7508170-184a-47cc-9319-c53b95dccd82/istio-proxy/1.log"
Sep 16 20:04:27.233828 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:27.233791    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j_d7508170-184a-47cc-9319-c53b95dccd82/istio-proxy/0.log"
Sep 16 20:04:27.250101 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:27.250078    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j_d7508170-184a-47cc-9319-c53b95dccd82/istio-proxy/1.log"
Sep 16 20:04:28.533643 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:28.533615    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-rf9nd_617e873b-6589-470a-8a0f-f7d17e403bf4/kuadrant-console-plugin/0.log"
Sep 16 20:04:28.690251 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:28.690220    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-v89xw_bdbc426f-12b9-481e-b689-58a0a7b8dc3c/limitador/0.log"
Sep 16 20:04:29.366488 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:29.366455    2570 generic.go:356] "Generic (PLEG): container finished" podID="d8c16109-e449-49aa-9a22-68b307fe6151" containerID="d437b341c24eb80b43bbe184ce2782b3d878515a88db5f9db0d90a9ce3a1062e" exitCode=0
Sep 16 20:04:29.366635 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:29.366506    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mht7h/must-gather-mc2nt" event={"ID":"d8c16109-e449-49aa-9a22-68b307fe6151","Type":"ContainerDied","Data":"d437b341c24eb80b43bbe184ce2782b3d878515a88db5f9db0d90a9ce3a1062e"}
Sep 16 20:04:29.366830 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:29.366804    2570 scope.go:117] "RemoveContainer" containerID="d437b341c24eb80b43bbe184ce2782b3d878515a88db5f9db0d90a9ce3a1062e"
Sep 16 20:04:30.286809 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:30.286778    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-mht7h_must-gather-mc2nt_d8c16109-e449-49aa-9a22-68b307fe6151/gather/0.log"
Sep 16 20:04:34.216183 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:34.216150    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-49tcg_b2e821ae-3c87-4fa1-8065-4004d791b3fc/global-pull-secret-syncer/0.log"
Sep 16 20:04:34.477435 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:34.477361    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mrbh7_58db8d7a-7827-440d-93c3-4ef40d2cab4b/konnectivity-agent/0.log"
Sep 16 20:04:34.570695 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:34.570668    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-244.ec2.internal_f223e05d53cb9f1d1ecb96b3f640ab21/haproxy/0.log"
Sep 16 20:04:35.803109 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:35.803073    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-mht7h/must-gather-mc2nt"]
Sep 16 20:04:35.803517 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:35.803248    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-mht7h/must-gather-mc2nt" podUID="d8c16109-e449-49aa-9a22-68b307fe6151" containerName="copy" containerID="cri-o://fa5148b2b0dff3c54f548132f3290027eba368526189c4a33cb966530ea4e20b" gracePeriod=2
Sep 16 20:04:35.808035 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:35.808005    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-mht7h/must-gather-mc2nt"]
Sep 16 20:04:36.030538 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.030515    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-mht7h_must-gather-mc2nt_d8c16109-e449-49aa-9a22-68b307fe6151/copy/0.log"
Sep 16 20:04:36.033114 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.033095    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mht7h/must-gather-mc2nt"
Sep 16 20:04:36.034996 ip-10-0-135-244 kubenswrapper[2570]: E0916 20:04:36.034973    2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-mc2nt\" is forbidden: User \"system:node:ip-10-0-135-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-mht7h\": no relationship found between node 'ip-10-0-135-244.ec2.internal' and this object" podUID="d8c16109-e449-49aa-9a22-68b307fe6151" pod="openshift-must-gather-mht7h/must-gather-mc2nt"
Sep 16 20:04:36.105884 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.105834    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvjq7\" (UniqueName: \"kubernetes.io/projected/d8c16109-e449-49aa-9a22-68b307fe6151-kube-api-access-mvjq7\") pod \"d8c16109-e449-49aa-9a22-68b307fe6151\" (UID: \"d8c16109-e449-49aa-9a22-68b307fe6151\") "
Sep 16 20:04:36.105884 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.105875    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8c16109-e449-49aa-9a22-68b307fe6151-must-gather-output\") pod \"d8c16109-e449-49aa-9a22-68b307fe6151\" (UID: \"d8c16109-e449-49aa-9a22-68b307fe6151\") "
Sep 16 20:04:36.107881 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.107854    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c16109-e449-49aa-9a22-68b307fe6151-kube-api-access-mvjq7" (OuterVolumeSpecName: "kube-api-access-mvjq7") pod "d8c16109-e449-49aa-9a22-68b307fe6151" (UID: "d8c16109-e449-49aa-9a22-68b307fe6151"). InnerVolumeSpecName "kube-api-access-mvjq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:04:36.113260 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.113238    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c16109-e449-49aa-9a22-68b307fe6151-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "d8c16109-e449-49aa-9a22-68b307fe6151" (UID: "d8c16109-e449-49aa-9a22-68b307fe6151"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:04:36.160848 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.160822    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8c16109-e449-49aa-9a22-68b307fe6151" path="/var/lib/kubelet/pods/d8c16109-e449-49aa-9a22-68b307fe6151/volumes"
Sep 16 20:04:36.206486 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.206464    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvjq7\" (UniqueName: \"kubernetes.io/projected/d8c16109-e449-49aa-9a22-68b307fe6151-kube-api-access-mvjq7\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:04:36.206486 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.206483    2570 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d8c16109-e449-49aa-9a22-68b307fe6151-must-gather-output\") on node \"ip-10-0-135-244.ec2.internal\" DevicePath \"\""
Sep 16 20:04:36.394065 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.394006    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-mht7h_must-gather-mc2nt_d8c16109-e449-49aa-9a22-68b307fe6151/copy/0.log"
Sep 16 20:04:36.394364 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.394344    2570 generic.go:356] "Generic (PLEG): container finished" podID="d8c16109-e449-49aa-9a22-68b307fe6151" containerID="fa5148b2b0dff3c54f548132f3290027eba368526189c4a33cb966530ea4e20b" exitCode=143
Sep 16 20:04:36.394415 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.394387    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mht7h/must-gather-mc2nt"
Sep 16 20:04:36.394415 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.394400    2570 scope.go:117] "RemoveContainer" containerID="fa5148b2b0dff3c54f548132f3290027eba368526189c4a33cb966530ea4e20b"
Sep 16 20:04:36.401607 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.401589    2570 scope.go:117] "RemoveContainer" containerID="d437b341c24eb80b43bbe184ce2782b3d878515a88db5f9db0d90a9ce3a1062e"
Sep 16 20:04:36.414849 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.414832    2570 scope.go:117] "RemoveContainer" containerID="fa5148b2b0dff3c54f548132f3290027eba368526189c4a33cb966530ea4e20b"
Sep 16 20:04:36.415100 ip-10-0-135-244 kubenswrapper[2570]: E0916 20:04:36.415078    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5148b2b0dff3c54f548132f3290027eba368526189c4a33cb966530ea4e20b\": container with ID starting with fa5148b2b0dff3c54f548132f3290027eba368526189c4a33cb966530ea4e20b not found: ID does not exist" containerID="fa5148b2b0dff3c54f548132f3290027eba368526189c4a33cb966530ea4e20b"
Sep 16 20:04:36.415164 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.415107    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5148b2b0dff3c54f548132f3290027eba368526189c4a33cb966530ea4e20b"} err="failed to get container status \"fa5148b2b0dff3c54f548132f3290027eba368526189c4a33cb966530ea4e20b\": rpc error: code = NotFound desc = could not find container \"fa5148b2b0dff3c54f548132f3290027eba368526189c4a33cb966530ea4e20b\": container with ID starting with fa5148b2b0dff3c54f548132f3290027eba368526189c4a33cb966530ea4e20b not found: ID does not exist"
Sep 16 20:04:36.415164 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.415124    2570 scope.go:117] "RemoveContainer" containerID="d437b341c24eb80b43bbe184ce2782b3d878515a88db5f9db0d90a9ce3a1062e"
Sep 16 20:04:36.415371 ip-10-0-135-244 kubenswrapper[2570]: E0916 20:04:36.415354    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d437b341c24eb80b43bbe184ce2782b3d878515a88db5f9db0d90a9ce3a1062e\": container with ID starting with d437b341c24eb80b43bbe184ce2782b3d878515a88db5f9db0d90a9ce3a1062e not found: ID does not exist" containerID="d437b341c24eb80b43bbe184ce2782b3d878515a88db5f9db0d90a9ce3a1062e"
Sep 16 20:04:36.415420 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:36.415375    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d437b341c24eb80b43bbe184ce2782b3d878515a88db5f9db0d90a9ce3a1062e"} err="failed to get container status \"d437b341c24eb80b43bbe184ce2782b3d878515a88db5f9db0d90a9ce3a1062e\": rpc error: code = NotFound desc = could not find container \"d437b341c24eb80b43bbe184ce2782b3d878515a88db5f9db0d90a9ce3a1062e\": container with ID starting with d437b341c24eb80b43bbe184ce2782b3d878515a88db5f9db0d90a9ce3a1062e not found: ID does not exist"
Sep 16 20:04:39.473224 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:39.473191    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-rf9nd_617e873b-6589-470a-8a0f-f7d17e403bf4/kuadrant-console-plugin/0.log"
Sep 16 20:04:39.673033 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:39.673008    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-v89xw_bdbc426f-12b9-481e-b689-58a0a7b8dc3c/limitador/0.log"
Sep 16 20:04:42.272925 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:42.272896    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-md657_9a9cce6a-0af1-4da9-abd9-a55db5f340e6/node-exporter/0.log"
Sep 16 20:04:42.306942 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:42.306921    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-md657_9a9cce6a-0af1-4da9-abd9-a55db5f340e6/kube-rbac-proxy/0.log"
Sep 16 20:04:42.363786 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:42.363762    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-md657_9a9cce6a-0af1-4da9-abd9-a55db5f340e6/init-textfile/0.log"
Sep 16 20:04:43.989334 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:43.989298    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"]
Sep 16 20:04:43.989753 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:43.989532    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8c16109-e449-49aa-9a22-68b307fe6151" containerName="gather"
Sep 16 20:04:43.989753 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:43.989543    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8c16109-e449-49aa-9a22-68b307fe6151" containerName="gather"
Sep 16 20:04:43.989753 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:43.989553    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8c16109-e449-49aa-9a22-68b307fe6151" containerName="copy"
Sep 16 20:04:43.989753 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:43.989559    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8c16109-e449-49aa-9a22-68b307fe6151" containerName="copy"
Sep 16 20:04:43.989753 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:43.989612    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8c16109-e449-49aa-9a22-68b307fe6151" containerName="copy"
Sep 16 20:04:43.989753 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:43.989620    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8c16109-e449-49aa-9a22-68b307fe6151" containerName="gather"
Sep 16 20:04:43.995468 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:43.995448    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:43.998094 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:43.998074    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-srgqk\"/\"default-dockercfg-tv7gj\""
Sep 16 20:04:43.998937 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:43.998916    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-srgqk\"/\"kube-root-ca.crt\""
Sep 16 20:04:43.999041 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:43.998956    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-srgqk\"/\"openshift-service-ca.crt\""
Sep 16 20:04:44.005093 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.005070    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"]
Sep 16 20:04:44.056758 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.056712    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhxks\" (UniqueName: \"kubernetes.io/projected/41bc6e60-b238-4e97-92bd-3bcc1903d998-kube-api-access-jhxks\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.056758 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.056748    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41bc6e60-b238-4e97-92bd-3bcc1903d998-podres\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.056880 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.056793    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41bc6e60-b238-4e97-92bd-3bcc1903d998-lib-modules\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.056880 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.056812    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41bc6e60-b238-4e97-92bd-3bcc1903d998-proc\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.056880 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.056833    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41bc6e60-b238-4e97-92bd-3bcc1903d998-sys\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.157285 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.157262    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxks\" (UniqueName: \"kubernetes.io/projected/41bc6e60-b238-4e97-92bd-3bcc1903d998-kube-api-access-jhxks\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.157381 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.157295    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41bc6e60-b238-4e97-92bd-3bcc1903d998-podres\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.157381 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.157319    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41bc6e60-b238-4e97-92bd-3bcc1903d998-lib-modules\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.157493 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.157413    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41bc6e60-b238-4e97-92bd-3bcc1903d998-proc\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.157493 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.157426    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41bc6e60-b238-4e97-92bd-3bcc1903d998-lib-modules\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.157493 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.157426    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41bc6e60-b238-4e97-92bd-3bcc1903d998-podres\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.157493 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.157473    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41bc6e60-b238-4e97-92bd-3bcc1903d998-proc\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.157493 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.157473    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41bc6e60-b238-4e97-92bd-3bcc1903d998-sys\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.157780 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.157509    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41bc6e60-b238-4e97-92bd-3bcc1903d998-sys\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.166333 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.166310    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhxks\" (UniqueName: \"kubernetes.io/projected/41bc6e60-b238-4e97-92bd-3bcc1903d998-kube-api-access-jhxks\") pod \"perf-node-gather-daemonset-878qq\" (UID: \"41bc6e60-b238-4e97-92bd-3bcc1903d998\") " pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.304491 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.304470    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:44.575456 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:44.575430    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"]
Sep 16 20:04:44.577778 ip-10-0-135-244 kubenswrapper[2570]: W0916 20:04:44.577745    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod41bc6e60_b238_4e97_92bd_3bcc1903d998.slice/crio-4046d0b80c51bfa4d04f06a630874e593b603269068ef1a11404d43e4c62fe42 WatchSource:0}: Error finding container 4046d0b80c51bfa4d04f06a630874e593b603269068ef1a11404d43e4c62fe42: Status 404 returned error can't find the container with id 4046d0b80c51bfa4d04f06a630874e593b603269068ef1a11404d43e4c62fe42
Sep 16 20:04:45.428109 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:45.428074    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq" event={"ID":"41bc6e60-b238-4e97-92bd-3bcc1903d998","Type":"ContainerStarted","Data":"e02f91743fd14e6af823ecc834c4a3cde26e6d5005457b1cf7548022d20a202a"}
Sep 16 20:04:45.428497 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:45.428158    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq" event={"ID":"41bc6e60-b238-4e97-92bd-3bcc1903d998","Type":"ContainerStarted","Data":"4046d0b80c51bfa4d04f06a630874e593b603269068ef1a11404d43e4c62fe42"}
Sep 16 20:04:45.428497 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:45.428181    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:45.447901 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:45.447853    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq" podStartSLOduration=2.447838568 podStartE2EDuration="2.447838568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:04:45.446000756 +0000 UTC m=+4811.830559980" watchObservedRunningTime="2026-09-16 20:04:45.447838568 +0000 UTC m=+4811.832397824"
Sep 16 20:04:47.248206 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:47.248174    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vmm77_e388636a-eb1f-4148-b7df-dd7bff98deed/dns/0.log"
Sep 16 20:04:47.308728 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:47.308701    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vmm77_e388636a-eb1f-4148-b7df-dd7bff98deed/kube-rbac-proxy/0.log"
Sep 16 20:04:47.469133 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:47.469099    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jsj4b_41b9e54d-41ed-4362-a99a-9998cba62a60/dns-node-resolver/0.log"
Sep 16 20:04:48.190138 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:48.190106    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mzfr8_41f91dbd-52af-482f-88fe-b09f0e5c48cc/node-ca/0.log"
Sep 16 20:04:49.407108 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:49.407075    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j_d7508170-184a-47cc-9319-c53b95dccd82/istio-proxy/0.log"
Sep 16 20:04:49.421419 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:49.421396    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j_d7508170-184a-47cc-9319-c53b95dccd82/istio-proxy/1.log"
Sep 16 20:04:50.091705 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:50.091645    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rfb8v_3733edd7-fc83-4df8-a976-80b9c7baba29/serve-healthcheck-canary/0.log"
Sep 16 20:04:50.799805 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:50.799776    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d9v42_78a486cb-98bc-47ab-9679-c1bc9e3fc69d/kube-rbac-proxy/0.log"
Sep 16 20:04:50.832463 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:50.832436    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d9v42_78a486cb-98bc-47ab-9679-c1bc9e3fc69d/exporter/0.log"
Sep 16 20:04:50.860369 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:50.860349    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d9v42_78a486cb-98bc-47ab-9679-c1bc9e3fc69d/extractor/0.log"
Sep 16 20:04:51.439894 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:51.439869    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-srgqk/perf-node-gather-daemonset-878qq"
Sep 16 20:04:54.161305 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:54.161270    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j_d7508170-184a-47cc-9319-c53b95dccd82/istio-proxy/0.log"
Sep 16 20:04:54.174316 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:54.174289    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j_d7508170-184a-47cc-9319-c53b95dccd82/istio-proxy/1.log"
Sep 16 20:04:54.572670 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:54.572644    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-796cdc484c-ngwdv_09d75669-3823-486c-961c-d9b0f3f499ac/manager/0.log"
Sep 16 20:04:55.361883 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:55.361853    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j_d7508170-184a-47cc-9319-c53b95dccd82/istio-proxy/0.log"
Sep 16 20:04:55.378246 ip-10-0-135-244 kubenswrapper[2570]: I0916 20:04:55.378219    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-6b6bcd64c-v6j7j_d7508170-184a-47cc-9319-c53b95dccd82/istio-proxy/1.log"